Re: Dependency Update

2019-05-22 Thread Aaron Markham
Thanks for doing a thorough look at the version ranges. I have this PR
[1] waiting for review that tries to pin graphviz and opencv, and it
updates CI as well as the docs that go on the website.
I think your updates would be beneficial in the docs that go on the
website and should also update CI. Is there a benefit to having them
as a readme in /tools? Doesn't this create extra maintenance with
these version numbers being in three places (website install
instructions, /tools folder, /ci folder)?

[1] https://github.com/apache/incubator-mxnet/pull/14987

On Wed, May 22, 2019 at 2:31 PM Qing Lan  wrote:
>
>
> Great work Jake! The content on CPU/GPU build instruction is really helpful.
>
> Thanks,
> Qing
>
> 
> From: Jake Lee 
> Sent: Wednesday, May 22, 2019 17:26
> To: dev@mxnet.incubator.apache.org
> Subject: Dependency Update
>
> Dear Community,
>
> I have been working on dependency udpate of MXNet. The goal is to upgrade
> the dependencies that have security vulnerability issues and make MXNet
> great again by being benefit from latest CUDA, cuDNN and NCCL software. I
> documented the process on PR<
> https://github.com/apache/incubator-mxnet/pull/15045>. Big thanks to Sheng
> Zha(szha), Dick Carter(DickJC123), Anirudh Subramanian(anirudh2290), Qing
> Lan(lanking520), Per Goncalves da Silva(perdasilva) for supporting me!
>
> Thanks,
> Jake


Re: [Discussion] Remove bundled llvm OpenMP

2019-05-22 Thread Aaron Markham
I reopened it for you.

On Wed, May 22, 2019, 05:25 Anton Chernov  wrote:

> I don't have necessary rights to reopen this PR.
>
> пн, 20 мая 2019 г. в 08:00, Pedro Larroy :
>
> > Hi Anton, Stas.
> >
> > Can we reopen this PR and get it merged as per the data collected by
> Stas?
> >
> > https://github.com/apache/incubator-mxnet/pull/12160
> >
> >
> >
> https://cwiki.apache.org/confluence/display/MXNET/Benchmarking+MXNet+with+different+OpenMP+implementations
> >
> > There are multiple issues that will be fixed by solving this problem.
> >
> >
> > Pedro
> >
> > On Tue, Feb 12, 2019 at 4:54 AM Anton Chernov 
> wrote:
> > >
> > > I would like to propose a possible alternative solution for
> > consideration.
> > >
> > > If keeping llvm OpenMP as a submodule is inevitable one could make
> > > following adjustments:
> > >
> > > Since compilers try to find their own OpenMP library implicitly, MXNet
> > > needs to ensure that only the bundled version is found. Therefore
> during
> > > the build and also during deployment this library has to provide
> symlinks
> > > for each possible compiler that would link to the built artifact ie.
> > >
> > > libiomp.so -> libgomp.so -> libomp.so
> > >
> > > The MKLML iomp would need to be hidden and removed as well.
> > >
> > > On Windows it would be a different story, but as can be seen [1]
> bundled
> > > OpenMP was not included in the Windows build anyway.
> > >
> > > Alternatively: always use iomp (with same symlinking trick though)
> > provided
> > > by MKLML distribution [2]. This potentially could work on Windows as
> > well.
> > >
> > > Best
> > > Anton
> > >
> > > [1]
> > >
> >
> https://github.com/apache/incubator-mxnet/blob/8a63bdecf2d9f12d34fe5874957ae4c867eb5f5b/CMakeLists.txt#L408-L410
> > > [2] https://github.com/intel/mkl-dnn/releases
> > >
> > > вт, 12 февр. 2019 г. в 11:22, Anton Chernov :
> > >
> > > > Recent benchmarking results have been published here [1]. Experiments
> > > > compare different OpenMP implementations as well as binaries compiled
> > with
> > > > different compilers including GCC, Clang and ICC.
> > > >
> > > > During experimentation another issues with mixing up libraries was
> > > > identified and described here [2].
> > > >
> > > > Best
> > > > Anton
> > > >
> > > > [1] https://cwiki.apache.org/confluence/x/2wclBg
> > > > [2]
> > > >
> >
> https://github.com/apache/incubator-mxnet/issues/14087#issuecomment-461734041
> > > >
> > > >
> > > > вс, 9 дек. 2018 г. в 16:28, Anton Chernov :
> > > >
> > > >> Hi Chris,
> > > >>
> > > >> Following up on the issue, are all things resolved in the
> discussion?
> > > >>
> > > >> If yes, I kindly ask you to reopen this PR and remove ‘requesting
> > > >> changes’ status:
> > > >> https://github.com/apache/incubator-mxnet/pull/12160
> > > >>
> > > >> Thank you.
> > > >>
> > > >>
> > > >> Best
> > > >> Anton
> > > >>
> > > >>
> > > >> вт, 27 нояб. 2018 г. в 17:15, Anton Chernov :
> > > >>
> > > >>> Another thing to take into consideration:
> > > >>>
> > > >>> All python artefacts that are created (PyPi) are built with make
> and
> > are
> > > >>> not using the bundled OpenMP library.
> > > >>>
> > > >>> One step for the switch to CMake to happen is the approval and
> > merging
> > > >>> of the mentioned PR:
> > > >>>
> > > >>> https://github.com/apache/incubator-mxnet/pull/12160
> > > >>>
> > > >>> If there are no other objections I kindly ask Chris Olivier to
> remove
> > > >>> his 'requesting changes' veto on it to unblock the CMake overhaul
> > work.
> > > >>>
> > > >>> Thank you.
> > > >>>
> > > >>> Best
> > > >>> Anton
> > > >>>
> > > >>> чт, 22 нояб. 2018 г. в 17:11, Anton Chernov :
> > > >>>
> > > 
> > >  Thank you for you answer, Chris.
> > > 
> > >  > The whole “mixing omp libraries” is something that occurs in
> > >  production
> > >  every day and certainly in everything that uses mkl.
> > > 
> > >  I'm afraid this statement is wrong. Intel MKL-DNN strictly ensures
> > that
> > >  this mixture is not happening:
> > > 
> > >  "Intel MKL-DNN uses OpenMP* for parallelism and requires an OpenMP
> > >  runtime library to work. As different OpenMP runtimes may not be
> > binary
> > >  compatible it's important to ensure that only one OpenMP runtime
> is
> > used
> > >  throughout the application. Having more than one OpenMP runtime
> > initialized
> > >  may lead to undefined behavior resulting in incorrect results or
> > crashes."
> > >  [1]
> > > 
> > >  That is why 2 different MKLML libraries are provided:
> > > 
> > >  lib/libmklml_gnu.so  | Intel MKL small library for GNU* OpenMP
> > runtime
> > >  lib/libmklml_intel.so | Intel MKL small library for Intel(R)
> OpenMP
> > >  runtime
> > > 
> > >  > is the suggestion that libiomp be removed from mkl?
> > > 
> > >  That is certainly not my suggestion.
> > > 
> > >  > have you spoken with intel? have you consulted Intel at all?
> > > 
> 

Re: Python2 End of Life

2019-05-13 Thread Aaron Markham
+1 for the pledge and to start moving things to Python 3.
I think our installation instructions and tutorials can be updated to
default to Python3 and we should update Python2-only tutorials. I know
we have a handful of those, and when I spot them, I'll create an
issue.
I can also look at migrating the docs build to Python 3.
Should we add a new label for issues relating to migrating to Python3?
Cheers,
Aaron

On Mon, May 13, 2019 at 12:04 PM Zach Kimberg  wrote:
>
> Right now, the official date for ending support for Python 2.7 (and all of
> python2) is set to January 1 [1]. As part of it, a number of projects have
> pledged to drop support for Python2 in or before 2020 including Tensorflow,
> requests, pandas, ipython, numpy, pillow, and Cython [2]. I believe we
> should also join in this pledge on python3statement.org [2] because it
> would help clean up our project and it would be difficult to continue
> supporting Python2 anyway when some of our dependencies are dropping
> support.
>
> As a concrete step, we should decide on a date to remove all usages of
> Python2 from our CI and consider that officially dropping support.
> Following that, we can expect PRs will end up breaking support for Python2.
> I suggest just using the same date that Python is dropping support of
> January 1. We may also need to update some examples or scripts that were
> written only for python2 that are around the project. Any thoughts?
>
> Zach
>
>
> [1] - https://www.python.org/dev/peps/pep-0373/
> [2] - https://python3statement.org/


Re: Unable to comment on GitHub issue

2019-05-09 Thread Aaron Markham
I just locked one of the issues I created:
https://github.com/apache/incubator-mxnet/issues/14918
Are you sure you don't have the unlock button on the right side?
You should see this:

aaronmarkham locked as off topic and limited conversation to
collaborators 24 seconds from now

Then to the right of that:

 Unlock conversation
 Pin issue

On Thu, May 9, 2019 at 4:27 PM Naveen Swamy  wrote:
>
> I don't see the option, another possible explanation someone must have 
> blocked me, if that is the case it goes against the ethos of Open source.
> Apache infra should override that setting for Apache projects. Anyway I 
> created this Jira.
> https://issues.apache.org/jira/plugins/servlet/mobile#issue/INFRA-18356
>
> -Naveen
>
> > On May 9, 2019, at 4:19 PM, Aaron Markham  wrote:
> >
> > A new feature: https://help.github.com/en/articles/locking-conversations
> > So someone must have locked it. I can see the option on the right hand
> > side column, all the way at the bottom. You will probably have the
> > ability to unlock it from there too.
> >
> >> On Thu, May 9, 2019 at 3:42 PM Chaitanya Bapat  
> >> wrote:
> >>
> >> Any specific issues you could give the links to? So I could verify if
> >> that's the case with me.
> >>
> >>> On Thu, 9 May 2019 at 14:44, Naveen Swamy  wrote:
> >>>
> >>> I am unable to comment on certain GitHub issues and see a locked Icon,
> >>> wondering if anyone has experienced this and know why?
> >>>
> >>
> >>
> >> --
> >> *Chaitanya Prakash Bapat*
> >> *+1 (973) 953-6299*
> >>
> >> [image: https://www.linkedin.com//in/chaibapat25]
> >> <https://github.com/ChaiBapchya>[image: https://www.facebook.com/chaibapat]
> >> <https://www.facebook.com/chaibapchya>[image:
> >> https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya>[image:
> >> https://www.linkedin.com//in/chaibapat25]
> >> <https://www.linkedin.com//in/chaibapchya/>


Re: Unable to comment on GitHub issue

2019-05-09 Thread Aaron Markham
A new feature: https://help.github.com/en/articles/locking-conversations
So someone must have locked it. I can see the option on the right hand
side column, all the way at the bottom. You will probably have the
ability to unlock it from there too.

On Thu, May 9, 2019 at 3:42 PM Chaitanya Bapat  wrote:
>
> Any specific issues you could give the links to? So I could verify if
> that's the case with me.
>
> On Thu, 9 May 2019 at 14:44, Naveen Swamy  wrote:
>
> > I am unable to comment on certain GitHub issues and see a locked Icon,
> > wondering if anyone has experienced this and know why?
> >
>
>
> --
> *Chaitanya Prakash Bapat*
> *+1 (973) 953-6299*
>
> [image: https://www.linkedin.com//in/chaibapat25]
> [image: https://www.facebook.com/chaibapat]
> [image:
> https://twitter.com/ChaiBapchya] [image:
> https://www.linkedin.com//in/chaibapat25]
> 


Re: [DISCUSS] AWS Credits for External Contributors

2019-05-09 Thread Aaron Markham
One option is Amazon Educate. https://aws.amazon.com/education/awseducate/
Last I checked, you can get $75/month AWS credit as a student or
educator. If you belong to an educational organization, your org can
apply on your behalf and get anyone with that org's domain easier
access to the credits. Or something like that.

Another route is you might be able to load your test/work into a
notebook and run it on Google Colab. Vandana has this neat DCGAN with
MXNet notebook running there.
https://colab.research.google.com/github/vandanavk/mxnet-gluon-gan/blob/dcgan/dcgan/dcgan.ipynb

Will either of those work for you?

Cheers,
Aaron

On Thu, May 9, 2019 at 11:30 AM Chaitanya Bapat  wrote:
>
> Hello MXNet community,
>
> I was curious to know if there is any possibility of AWS Credits
> provisioned for external contributors of Apache MXNet. It would be a great
> incentive for more external contributions and in turn more external
> contributors.
>
> Background -
> Today, while trying to work on Anirudh's Memory profiling for MXNet PR, I
> realized I am short of AWS credits on my personal account. My personal
> computer (Mac 2017) doesn't have Nvidia GPU and hence I'm a bit stuck.
>
> I don't know if there are others who have faced a similar situation. If
> that's the case, maybe we can find a solution through free AWS Credits.
>
> Thanks,
> Chai
>
> --
> *Chaitanya Prakash Bapat*
> *+1 (973) 953-6299*
>
> [image: https://www.linkedin.com//in/chaibapat25]
> [image: https://www.facebook.com/chaibapat]
> [image:
> https://twitter.com/ChaiBapchya] [image:
> https://www.linkedin.com//in/chaibapat25]
> 


Re: [VOTE] Release Apache MXNet (incubating) version 1.4.1.rc0

2019-05-01 Thread Aaron Markham
Make that +1 (non-binding)

On Wed, May 1, 2019 at 3:42 PM Aaron Markham  wrote:
>
> +1 (binding)
>
> * Built with GPU and tested the first part of the ssd example.
> * Built with GPU / cross-compiled to arm8 for Jetson.
> * Built Scala/Java on top of the cross-compiled arm8 (ran into trouble
> here, but I think this is not popular enough yet to derail things,
> plus there are workarounds)
> * Built on CPU instance and tested docs.
> http://34.201.8.176/versions/1.4.1/api/python/io/io.html
> I don't see anything specific being different in this patch for docs,
> so hard to tell if there's an issue. I'll assume not given the
> successful generation of the API docs.
>
>
> On Wed, May 1, 2019 at 1:28 PM Pedro Larroy
>  wrote:
> >
> > +1 (non-binding)
> >
> > Tried CPU build + C++ tests + 714 Python unit tests in 605s.
> > ARMv7 build + small unit test in QEMU + ARMv8 builds.
> >
> > Thanks. Regards
> >
> > Pedro.
> >
> > On Wed, May 1, 2019 at 10:41 AM Qing Lan  wrote:
> > >
> > > +1 (binding)
> > >
> > > build from source works for OSX and Ubuntu CPU
> > > Scala build/test successfully with Dynamic link and static link.
> > >
> > > Thanks,
> > > Qing
> > >
> > > 
> > > From: Sheng Zha 
> > > Sent: Wednesday, May 1, 2019 13:14
> > > To: d...@mxnet.apache.org
> > > Subject: Re: [VOTE] Release Apache MXNet (incubating) version 1.4.1.rc0
> > >
> > > Hi all,
> > >
> > > Reminder that the vote for 1.4.1 release is still ongoing. If you can, 
> > > please help out. Thank you.
> > >
> > > -sz
> > >
> > > On 2019/04/30 06:51:45, Junru Shao  wrote:
> > > > Dear MXNet community,
> > > >
> > > > This is the 3-day vote to release Apache MXNet (incubating) version 
> > > > v1.4.1.
> > > > The voting on dev@ list will start Apr 29 23:59:59 (PST) and close on 
> > > > May
> > > > 02 23:59:59.
> > > >
> > > > Below are links to
> > > > 1) Release notes:
> > > > https://cwiki.apache.org/confluence/display/MXNET/Apache+MXNet+%28incubating%29+1.4.1+Release+Notes
> > > > .
> > > > 2) Release Candidate:
> > > > https://github.com/apache/incubator-mxnet/releases/tag/1.4.1.rc0.
> > > > 3) Source and signatures on Apache dist server:
> > > > https://dist.apache.org/repos/dist/dev/incubator/mxnet/1.4.1.rc0/.
> > > >
> > > > Please remember to TEST first before voting accordingly:
> > > > +1 = approve
> > > > +0 = no opinion
> > > > -1 = disapprove (provide reason)
> > > >
> > > > Best regards,
> > > > Junru Shao
> > > >


Re: [VOTE] Release Apache MXNet (incubating) version 1.4.1.rc0

2019-05-01 Thread Aaron Markham
+1 (binding)

* Built with GPU and tested the first part of the ssd example.
* Built with GPU / cross-compiled to arm8 for Jetson.
* Built Scala/Java on top of the cross-compiled arm8 (ran into trouble
here, but I think this is not popular enough yet to derail things,
plus there are workarounds)
* Built on CPU instance and tested docs.
http://34.201.8.176/versions/1.4.1/api/python/io/io.html
I don't see anything specific being different in this patch for docs,
so hard to tell if there's an issue. I'll assume not given the
successful generation of the API docs.


On Wed, May 1, 2019 at 1:28 PM Pedro Larroy
 wrote:
>
> +1 (non-binding)
>
> Tried CPU build + C++ tests + 714 Python unit tests in 605s.
> ARMv7 build + small unit test in QEMU + ARMv8 builds.
>
> Thanks. Regards
>
> Pedro.
>
> On Wed, May 1, 2019 at 10:41 AM Qing Lan  wrote:
> >
> > +1 (binding)
> >
> > build from source works for OSX and Ubuntu CPU
> > Scala build/test successfully with Dynamic link and static link.
> >
> > Thanks,
> > Qing
> >
> > 
> > From: Sheng Zha 
> > Sent: Wednesday, May 1, 2019 13:14
> > To: d...@mxnet.apache.org
> > Subject: Re: [VOTE] Release Apache MXNet (incubating) version 1.4.1.rc0
> >
> > Hi all,
> >
> > Reminder that the vote for 1.4.1 release is still ongoing. If you can, 
> > please help out. Thank you.
> >
> > -sz
> >
> > On 2019/04/30 06:51:45, Junru Shao  wrote:
> > > Dear MXNet community,
> > >
> > > This is the 3-day vote to release Apache MXNet (incubating) version 
> > > v1.4.1.
> > > The voting on dev@ list will start Apr 29 23:59:59 (PST) and close on May
> > > 02 23:59:59.
> > >
> > > Below are links to
> > > 1) Release notes:
> > > https://cwiki.apache.org/confluence/display/MXNET/Apache+MXNet+%28incubating%29+1.4.1+Release+Notes
> > > .
> > > 2) Release Candidate:
> > > https://github.com/apache/incubator-mxnet/releases/tag/1.4.1.rc0.
> > > 3) Source and signatures on Apache dist server:
> > > https://dist.apache.org/repos/dist/dev/incubator/mxnet/1.4.1.rc0/.
> > >
> > > Please remember to TEST first before voting accordingly:
> > > +1 = approve
> > > +0 = no opinion
> > > -1 = disapprove (provide reason)
> > >
> > > Best regards,
> > > Junru Shao
> > >


docs updates

2019-04-19 Thread Aaron Markham
Hello everyone!
I've been pecking away at adding content to the beta site. Take a look
and let me know what you think. Right now, I'm focused 100% on the
content and the organization thereof.

One feature I thought would be great for the API docs would be to
cross-reference usage of a particular function with an example or
tutorial. As I researched approaches I found sphinx-gallery [2]
already supports this.

When trying to apply this to some examples we have I ran across an
issue [3] that many examples don't run out of the box. This prevents
sphinx-gallery from doing its magic.

My question to you all is: what you think of having .py files as a
primary source of example usage and to generate notebooks and web
pages from those. This is as opposed to .md files that get converted
to notebooks and then html. GluonCV's site uses this and it seems to
work pretty well for generating the tutorial web pages from the .py
files. The implication is fixing many examples to run when executed by
Sphinx. The remainder would have to be excluded and wouldn't be a
source of example usage in the docs.

Cheers,
Aaron

[1] https://beta.mxnet.io/
[2] 
https://sphinx-gallery.github.io/configuration.html#auto-documenting-your-api-with-links-to-examples
[3] https://github.com/apache/incubator-mxnet/issues/5717


Re: Implementing zero-dim and zero-size tensors in MXNet and its impact on your codebases

2019-04-11 Thread Aaron Markham
Just curious about when this kind of change will land. Would it wait for
2.0 or would it be in 1.5 or another minor release?

On Thu, Apr 11, 2019, 00:15 Junru Shao  wrote:

> Really nice improvement over MXNet's usability! I suggest that we could
> make numpy-compatible behavior default in 2.0.
>
> On Wed, Apr 10, 2019 at 11:34 PM Jun Wu  wrote:
>
> > Dear Community,
> >
> > A while ago, we sent out an RFC
> >  discussing the
> > initiative introducing NumPy compatibility into MXNet. As the first
> outcome
> > of this initiative, we submitted the PR
> >  providing the
> > infrastructure of supporting zero-dim (scalar) and zero-size tensors,
> which
> > have been long-missing in MXNet.
> >
> > In our implementation, we have put the best efforts of keeping the
> promise
> > of backward compatibility in all the language bindings. Nevertheless, we
> > still would like to call out the changes explicitly that may impact your
> > existing codebases developed on top of MXNet by calling C-APIs directly
> or
> > implementing operators in your own repos.
> >
> > 1. In you application, if you called any one of the following
> shape-related
> > C-APIs, you will need to change the data type of shape's ndim and
> dim_size
> > from *unsigned int* to signed *int*, because we have to use -1 to
> represent
> > unknown shape information, and reserve 0 for scalar and zero-size
> tensors.
> > One example of such changes can be seen in the cpp-package
> > <
> >
> https://github.com/apache/incubator-mxnet/pull/14661/files#diff-c0e1fcfe1619faa4ff5f59d94e8bR183
> > >
> > calling MXSymbolInferShape.
> > - MXSymbolInfershape
> > - MXSymbolInfershapePartial
> > - MXExecutorSimpleBind
> > - MXExecutorReshape
> > - MXNDArrayGetShape
> > - MXNDArrayCreaetFromSharedMem
> >
> > 2. If you have implemented operators in your own codebases, you will
> > probably need to change every operator's shape inference function to use
> > the following util functions to check whether shape information is known,
> > instead of checking against 0 directly. One example of such changes can
> be
> > seen in the shape inference function
> > <
> >
> https://github.com/apache/incubator-mxnet/pull/14661/files#diff-afa640c4653c59f00f43a84455f91ef9R35
> > >
> > of concat operator.
> > - shape_is_known (include/mxnet/tuple.h)
> > - ndim_is_known (include/mxnet/tuple.h)
> > - dim_size_is_known (include/mxnet/tuple.h)
> >
> > If you are interested in knowing the value of scalar tensors, and hence
> > understanding our motivation further, this thread
> > 
> of
> > discussion provides very good insights from the view of data science. It
> > was actually related to an opportunity for MXNet becoming the backend of
> > PyMC , but somehow it didn't go
> > through due to missing several key features
> > , and
> > scalar tensors is one of them.
> >
> > Please leave comments in the PR
> >  if you have any
> > concerns or suggestions of our work.
> >
> > Thank you very much for your time and consideration.
> >
> > Best,
> > Jun
> >
> > *References*
> > [1] RFC of NumPy compatibility:
> > https://github.com/apache/incubator-mxnet/issues/14253
> > [2] Pull request of supporting scalar and zero-size tensors:
> > https://github.com/apache/incubator-mxnet/pull/14661
> > [3] The value of scalar tensors from the view of data science:
> > https://discuss.mxnet.io/t/rank-0-arrays-in-mxnet-aka-pi-is-wrong/108
> > [4] Previous discussion for MXNet becoming the backend of PyMC:
> > https://discuss.mxnet.io/t/moving-pymc3-from-theano-to-mxnet/86
> >
>


Re: assimilation of mshadow into the MXNet codebase

2019-04-07 Thread Aaron Markham
+1
Reduced complexity. Choice of math library... Hopefully you can just
install MKL and not be forced into mshadow's dependency on OpenBLAS. This
could make Windows setup easier.
Maybe this issue will get fixed: #11769.

On Sun, Apr 7, 2019, 00:51 Junru Shao  wrote:

> Does merging mshadow into mxnet bring any actual benefit for customers in
> sense of performance, portability, or anything else?
>
> On Fri, Apr 5, 2019 at 9:38 PM Tianqi Chen 
> wrote:
>
> > Technically, mshadow is sufficient for MXNet. Adopting other libraries (
> > eigen or xtensor) will unnecessarily increase the codebase complexity
> > without any additional gains.
> >
> > Given that mshadow is only used by mxnet. I do support donating it into
> > mxnet codebase.
> > To respect the original mshadow community. I would recommend starting a
> > community RFC In the mshadow github issue for a week, before we start the
> > migrating process.
> > Also, I would recommend a rebase merge just like the case of MXNet.jl
> code
> > base to preserve the contribution history.
> >
> > Tianqi
> >
> >
> > On Fri, Apr 5, 2019 at 9:25 PM Alfredo Luque
> >  wrote:
> >
> > > Do you have a link to both of these proposals?
> > >
> > > On Fri, Apr 5, 2019 at 20:14 Anirudh Acharya 
> > > wrote:
> > >
> > > > Hi Pedro,
> > > >
> > > > mshadow is mostly used for tensor arithmetic. There have been
> > discussions
> > > > about including it within mxnet. I think it is a good idea.
> > > >
> > > > As a more long term solution using libraries like eigen to perform
> > linear
> > > > algebra operations was also suggested by anirudh2290@. I think
> > xtensor(
> > > > https://github.com/QuantStack/xtensor ) can also be a candidate
> here.
> > > >
> > > > -
> > > > Anirudh
> > > >
> > > >
> > > > On Fri, Apr 5, 2019 at 7:03 PM Pedro Larroy <
> > > pedro.larroy.li...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi
> > > > >
> > > > > Some developers have noticed that working in mshadow is cumbersome
> as
> > > > > it's a 3rdparty subrepo.
> > > > >
> > > > > Since mshadow is a bunch of headers which don't have much of
> > > > > independent tests / library functionality, me and other developers
> > > > > believe that it would be good to assimilate this code in the
> > > > > repository for ease of contribution and changes without having to
> go
> > > > > trough contortions to test PRs that modify mshadow.
> > > > >
> > > > > Would anybody oppose this change?
> > > > >
> > > > > Thanks and have a nice weekend.
> > > > >
> > > > > Pedro.
> > > > >
> > > >
> > >
> >
>


Re: [DISCUSS] Rebrand Gluon to MXNet imperative or something MXNet.

2019-03-28 Thread Aaron Markham
I am disappointed when I see articles that use Gluon, but don't mention
MXNet [1]. If MSFT or anyone had actually run with it and made Gluon an
interface for another framework, this would be a different matter. There
are more references to Gluon without MXNet [2] than there are to Gluon with
MXNet [3].

Of course anyone can build something on MXNet and name it what they want
and decide not to mention MXNet. It's what makes for a great open source
project. But, if Gluon gets so much highlighting in the project and on the
MXNet website, then it impacts the project's "brand".

When people have a CV or NLP need it would be great if they thought about
MXNet to solve this. I think it's important to keep the project's name in
the mix. People will abbreviate and settle on a term, and maybe one day
that'll be Gluon. But let's focus and be flexible for future contributions.
This is a long-winded way to say we can't rebrand someone's outside project
(like GluonCV), but if they'd like to contribute it to the project, we may
rebrand it to prevent confusion and improve awareness of the MXNet project.
Aside from that, anything already in the project, like the Python Gluon
API, should have MXNet in the name.

Would it be fair to say (and get confirmation from Apache) that we (the
community) can use companion terms with MXNet for the products of our
collaboration?
For example, a modifier like MXNetDoodles would not be okay as it impinges
the trademark directly. But these might be fine:
* MXNet Gluon
* MXNet CV (notice the space, not MXNetCV)
* MXNet NLP
* MXNet GluonCV
I would personally like to see more "use case" toolkits, so it's less of a
mouthful. Saying MXNet CV is easier than MXNet GluonCV, and it's less
confusing. You can say "MXNet CV with Gluon", and it is clear in its
function and its domain. So what if the "rebrand" is using descriptors like
"with" or "in"?
MXNet in Gluon. MXNet Python in imperative Gluon. That's a mouthful too,
but at least the reader can tell what's going and what it might be.

Whatever the case may be, +1 to asking that MXNet always be mentioned with
Gluon, to the point of making the term be MXNet Gluon. (Or some variant
thereof that I just described.)

[1] https://arxiv.org/pdf/1812.01187.pdf
[2]
https://www.google.com/search?q=deep+learning+gluon+-mxnet+-physics=deep+learning+gluon+-mxnet+-physics
[3]
https://www.google.com/search?q=deep+learning+gluon+mxnet+-physics=deep+learning+gluon+mxnet+-physics

On Thu, Mar 28, 2019 at 2:08 PM Mu Li  wrote:

> The name Gluon is originally used for a collaboration project between
> Amazon and Microsoft [1].
> I pinged both Apache and Amazon legal teams later, they confirmed Gluon is
> not considered as a trademark.
>
> [1]
>
> https://aws.amazon.com/blogs/aws/introducing-gluon-a-new-library-for-machine-learning-from-aws-and-microsoft/
>
> On Thu, Mar 28, 2019 at 2:04 PM Isabel Drost-Fromm 
> wrote:
>
> >
> >
> > Am 28. März 2019 21:53:16 MEZ schrieb Mu Li :
> > >
> > >The reason why we call it GluonCV instead of MXNetCV is because MXNet
> > >is a
> > >trademark owned by Apache, while Gluon doesn't have this issue.
> >
> > Who's the "we" in that sentence?
> >
> > If it doesn't belong to Apache, who owns the Gluon trademark?
> >
> > Isabel
> >
> >
> > --
> > Diese Nachricht wurde von meinem Android-Gerät mit K-9 Mail gesendet.
> >
>


Re: Call for Ideas and Approaches to Community Building

2019-03-20 Thread Aaron Markham
Anton, can you share the design and specs and code for the robot arm demo?
I wish that was being shown at GTC now. It would be great to let people
borrow it for West coast events. Maybe I can get one built here in Palo
Alto.

On Tue, Mar 19, 2019, 05:54 Anton Chernov  wrote:

> I don't know whether that is enough, but here are a few efforts we make to
> promote MXNet:
>
> * The robotic arms demo at the embedded world
> We promoted MXNet as the framework to go on embedded devices with our
> robotic arms demo. We've got a lot of attention from different people
> including professors from multiple universities. A blog post about the demo
> will be posted in the next days MXNet Medium blog [1].
>
> Here again some impressions from twitter:
> https://twitter.com/lebegus/status/1100839414228500485
>
> * MLPerf results
> We intend to publish more benchmark results to MLPerf [2], showing proof of
> the performance advantages of MXNet.
>
> * Recurring user group meetings
> We offer recurring VC meetings [3], free for everyone. We dedicate our time
> to anyone that would like to know more about MXNet or to ask any other
> related question.
>
> * Collaborative meetups
> We organize meetups with attendants from various companies [4], sharing
> their interesting use cases and best practises with ML and MXNet.
>
> Tracking works and papers on popular science conferences is a valid metric,
> but it's focused on research. More and more people that don't write papers
> use ML and MXNet in production without knowing all the scientific details.
> How to measure how many are out there is an open question.
>
> Best
> Anton
>
> [1] https://medium.com/apache-mxnet
> [2] https://mlperf.org/
> [3] https://cwiki.apache.org/confluence/x/7BY0BQ
> [4] https://www.meetup.com/Deep-Learning-with-Apache-MXNet-Berlin
>
>
> вт, 19 мар. 2019 г. в 07:23, Isabel Drost-Fromm :
>
> >
> >
> > Am 19. März 2019 02:49:23 MEZ schrieb "Zhao, Patric" <
> > patric.z...@intel.com>:
> > >I suggest to encourage and fund the students/researchers to present
> > >their works on the popular conference.
> > >I know talking is easy but maybe the decision maker can allocate more
> > >resources for marketing.
> >
> > Just for clarity, who exactly do you mean with "the decision maker"?
> > Decision maker for what?
> >
> > On another note, beyond that one conference, which other channels do
> > people here follow? How did you first hear about mxnet?
> >
> >
> > Isabel
> >
> >
> > --
> > Diese Nachricht wurde von meinem Android-Gerät mit K-9 Mail gesendet.
> >
>


Re: Requesting slack access

2019-03-14 Thread Aaron Markham
I sent you an invite. Welcome!

On Thu, Mar 14, 2019 at 7:24 AM Ge  wrote:

> thanks.
>


Re: [RFC] Integrating the new MXNet website

2019-03-11 Thread Aaron Markham
Weird - wonder why the dot was removed...
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103089084

On Mon, Mar 11, 2019 at 7:33 AM Anton Chernov  wrote:

> The link is not working:
> https://cwiki.apache.org/confluence/pages/viewpageaction?pageId=103089084
>
> 
>
> Page Not Found
> We can't find that page. This could be because:
>
> The page doesn't exist.
> The page exists, but you don't have view permission for that space.
>
> 
>
> I have checked that I'm logged in.
>
> Best
> Anton
>
> чт, 7 мар. 2019 г. в 20:09, Aaron Markham :
>
> > Thanks Aston.
> >
> > For contributors interested in helping improve the website's UX and
> design:
> > I spent some time learning the Material Design ecosystem as the beta
> site
> > uses this as its template. From there I refactored the wireframes using
> > Sketch + Material Design plugin and added them to gallery.io. These new
> > ones are for a typical user flow: they go from the Docs main page to
> Python
> > to Gluon (mxnet.gluon) and finally to mxnet.gluon.data. I put these
> > screenshots in the Website Redesign Information Architecture & Wireframe
> > wiki article (that I moved to the Proposals section):
> >
> https://cwiki.apache.org/confluence/pages/viewpageaction?pageId=103089084
> > Anyone that is interested can contribute directly to commenting on the
> > Material Design wireframe revisions here (you can also see drafts for
> > Ecosystem and Features):
> >
> >
> https://gallery.io/projects/MCHbtQVoQ2HCZfrqGyueCUJz/files/MCEJu8Y2hyDScX1HssyoxcEAyxZ6gioeNE8
> >
> > Cheers,
> > Aaron
> >
> >
> > On Mon, Mar 4, 2019 at 6:03 PM Aston Zhang  wrote:
> >
> > > Dear Community,
> > >
> > > We have published a post at
> > > https://github.com/apache/incubator-mxnet/issues/14330 requesting for
> > > comments on our proposal of integrating the new MXNet website. We'd
> like
> > > to hear
> > > the thoughts and suggestions from the community and welcome any form of
> > > contribution to improve contents and UX of the MXNet website.
> > >
> > > Thanks,
> > > MXNet developers from AWS
> > >
> >
>


Re: MXNet Community Monthly Updates

2019-03-08 Thread Aaron Markham
This is great. It will help drive traffic to the site, create content for
the site, and increase engagement. MailChimp is pretty good too. I can help
set that up. I have one tiny nagging question though... I don't recall
doing any GDPR compliance stuff on the site, so I'm wondering what kind of
EULA or popup thing we might need for the cookies and emails that come
along with this feature. Did anyone get blessed with this task recently and
know what needs to be done?

On Fri, Mar 8, 2019, 07:13 Carin Meier  wrote:

> I strongly support this as well - Let us know when the wiki draft page is
> ready. The Clojure MXNet contributors have some cool stuff to share :)
>
> On Thu, Mar 7, 2019 at 1:49 AM Junru Shao  wrote:
>
> > Hi Mu,
> >
> > Thanks for proposing this! Strongly agree with the newsletter - this
> would
> > be the most economically efficient way to enhance community involvement.
> >
> > In addition to the draft wiki pages, should we send posts like "Call for
> > newsletter articles" to dev@ list with some frequency? This could help
> > maximize the community awareness and help active community members get
> more
> > exposure - Community building is important.
> >
> > Thanks,
> > Junru
> >
> > On Wed, Mar 6, 2019 at 6:57 PM Chaitanya Bapat 
> > wrote:
> >
> > > Hello Mu,
> > >
> > > Thanks a lot for bringing this topic. I had thought about a Weekly
> Digest
> > > for MXNet (weekly newsletter) - which is on similar lines (can be made
> > into
> > > Monthly if it sounds good).
> > >
> > > Here's the quip doc -
> > > https://chaitanya.quip.com/BT6RAcAigHM9/MXNet-Weekly-Digest
> > >
> > > I have talked about the Background, Motivation, Features and a mockup
> of
> > > Newsletter.
> > >
> > > Would love to hear our community's thoughts as well as yours on the
> same.
> > >
> > > Please find attached a snapshot of the Weekly digest I came up with.
> > >
> > > Thanks,
> > > Chai
> > >
> > >
> > >
> > >
> > > On Wed, 6 Mar 2019 at 19:59, Mu Li  wrote:
> > >
> > >> Dear Community,
> > >>
> > >> I propose to send a monthly summary to users to broadcast the recent
> > >> progresses in the community. It will not only include new features
> added
> > >> into MXNet, but also various community activities. Here is an example:
> > >>
> > >> Tutorials
> > >> - 10 new lectures teaching at UC Berkeley
> > >> - Video record for "Deploying with Java" at Java World 19
> > >> Computer Vision
> > >> - GluonCV 0.4 release supports pose estimation and improves 10
> existing
> > >> models
> > >> - Insightface added a new model XY
> > >> NLP
> > >> - GluonNLP 0.5.1 release improves BERT training
> > >> New Projects
> > >> - A MXNet implementation for paper XY
> > >> MXNet
> > >> - Enhanced Java binding preview
> > >> - Numpy frontend reaches milestone 1
> > >> Incoming Events
> > >> - Meetup at Palto Alto on 4/2
> > >>
> > >> The publishing procedure is we first create a draft wiki page so
> > everyone
> > >> will have a chance to review and add staffs. After that we will send
> it
> > >> through an email list.
> > >>
> > >> I'm considering to use a 3rd party service such as mailchimp.com so
> > that
> > >> every user can subscribe it easily and we can do some marketing
> analysis
> > >> as
> > >> well. But I'm happy to re-use us...@mxnet.apache.org if it provides
> > >> simliar
> > >> functionalities.
> > >>
> > >> I'd like to hear your feedback for how to make the newletter more user
> > >> friendely.
> > >>
> > >> Best
> > >> Mu
> > >>
> > >
> > >
> > > --
> > > *Chaitanya Prakash Bapat*
> > > *+1 (973) 953-6299*
> > >
> > > [image: https://www.linkedin.com//in/chaibapat25]
> > > [image:
> > > https://www.facebook.com/chaibapat] <
> > https://www.facebook.com/chaibapchya>[image:
> > > https://twitter.com/ChaiBapchya]  > >[image:
> > > https://www.linkedin.com//in/chaibapat25]
> > > 
> > >
> >
>


Re: [RFC] Integrating the new MXNet website

2019-03-07 Thread Aaron Markham
Thanks Aston.

For contributors interested in helping improve the website's UX and design:
I spent some time learning the Material Design ecosystem as the beta site
uses this as its template. From there I refactored the wireframes using
Sketch + Material Design plugin and added them to gallery.io. These new
ones are for a typical user flow: they go from the Docs main page to Python
to Gluon (mxnet.gluon) and finally to mxnet.gluon.data. I put these
screenshots in the Website Redesign Information Architecture & Wireframe
wiki article (that I moved to the Proposals section):
https://cwiki.apache.org/confluence/pages/viewpageaction?pageId=103089084
Anyone that is interested can contribute directly to commenting on the
Material Design wireframe revisions here (you can also see drafts for
Ecosystem and Features):
https://gallery.io/projects/MCHbtQVoQ2HCZfrqGyueCUJz/files/MCEJu8Y2hyDScX1HssyoxcEAyxZ6gioeNE8

Cheers,
Aaron


On Mon, Mar 4, 2019 at 6:03 PM Aston Zhang  wrote:

> Dear Community,
>
> We have published a post at
> https://github.com/apache/incubator-mxnet/issues/14330 requesting for
> comments on our proposal of integrating the new MXNet website. We'd like
> to hear
> the thoughts and suggestions from the community and welcome any form of
> contribution to improve contents and UX of the MXNet website.
>
> Thanks,
> MXNet developers from AWS
>


Re: direction for documentation across various APIs that share common doc source

2019-03-06 Thread Aaron Markham
Mu,
Thanks for your response. I have some follow-up questions now. A lot actually.
Can you explain more about what ndarray_doc.py is doing? I see that
ndarray.register is calling it to do some transformations to
docstrings by injecting "float". This seems quite buried to me. Some
may have wondered tracing through the ndarray docs, "where does float
come from? It's not listed here in the docstring. Strange."
Is there a document that describes this pattern so other developers
know how to use it and what impact it has? Why am I not seeing the
pattern other than ndarray and symbol and only for float support? Why
doesn't symbol have a correlating symbol_doc.py? This makes me wonder
about the various issues I've seen where functions are properly
described, or at all, when Sphinx runs. Is this something that should
have been applied more widely, but has not? Wouldn't it make sense to
have the docs massaging processes centralized for maintenance and
clarity?

Aside from that, I'm still not seeing the path for solving the issue
with R and Scala and Java showing psuedocode or python code in their
examples by using `make doctest`. Maybe they're first steps to make
sure Python examples execute, but don't extend a solution to any other
language binding? That's fine if so, but I still want to keep
exploring what we do to facilitate good docs for the other bindings.
Are you perhaps suggesting that each language binding follow this
rewriting of the docstrings pattern that's in ndarray and symbol?

Can you look at this PR and provide feedback on a tangible example of
how to proceed? https://github.com/apache/incubator-mxnet/pull/14243



Vishaal & Anton, thanks for your feedback too. Flagging the code makes
a lot of sense as then it would be quite apparent what its intended
language is. Rerunning sphinx to rewrite the output could work, and
that assumes those packages have something specific and relevant to
inject. Unfortunately for R, Scala, and Java, that doesn't seem to be
the case as this point. Please correct me if I'm missing something
here.



Cheers,
Aaron

On Tue, Mar 5, 2019 at 9:44 AM Mu Li  wrote:
>
> The original design is putting psudo-code in cc files  (e.g. ndarray.cc
> <https://github.com/apache/incubator-mxnet/blob/master/src/ndarray/ndarray.cc>)
> that are languange indepent, then having python codes in .py files (e.g.
> ndarray_doc.py
> <https://github.com/apache/incubator-mxnet/blob/master/python/mxnet/ndarray_doc.py>).
> However, we haven't define the psudo-code format so some codes in cc files
> look like python, and we didn't enable doctest so some py file codes cannot
> be executed.
>
> I suggest the following next steps:
>
> 1. follow tensorflow's psudo-code format, e.g.
> https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/fill
> 2. enable doctest during building the doc (make doctest)
>
> On Mon, Mar 4, 2019 at 10:09 AM Vishaal Kapoor 
> wrote:
>
> > Hey Aaron and  Anton,
> >
> > One of MXNet's strengths over other frameworks is the plethora of language
> > bindings so having language specific examples is of importance. Perhaps
> > indicating that an example is Python code by using a "#python" header on
> > the example would make it clear.  Of course, for the important APIs,
> > docstrings for the most popular languages would be desired. Additionally,
> > making the holes clear would make it easier for users to contribute
> > documentation for their favorite languages.
> >
> > Vishaal
> >
> > On Mon, Mar 4, 2019 at 8:34 AM Anton Chernov  wrote:
> >
> > > Hi Aaron,
> > >
> > > Here is an idea: The main documentation is the one in .cc files. In
> > theory
> > > the language bindings should just override some stuff from it, like
> > > examples. If I understand correctly there is a sphinx script that
> > generates
> > > the documentation. If run it first for core src folder and then from a
> > > language binding folder it could use the -f, --force flag [1] to override
> > > the needed parts. That would allow to provide a 'default' version of the
> > > documentation, that could be adjusted where needed.
> > >
> > > Best
> > > Anton
> > >
> > > [1]
> > >
> > >
> > http://www.sphinx-doc.org/en/stable/man/sphinx-apidoc.html#sphinx-apidoc-manual-page
> > >
> > > вт, 26 февр. 2019 г. в 02:20, Aaron Markham :
> > >
> > > > Hi everyone,
> > > > A recent issue and pending PR has brought a thorny docs situation to
> > > > my attention again and I'd like to hear from the community on how to
> > > > proceed.
> > > > We currently get some of the docs for the 

Re: Call for Ideas and Approaches to Community Building

2019-03-06 Thread Aaron Markham
I'd like to see if we can get designers involved. I noted that there are
some boot camps for UX and design. They pick up projects for class to
design a brand or a new site or service. Why not get them involved with
open source? Apache has a lot of projects with sites that could use help,
and ours, especially the new one could use help.

I think designers would benefit as they'd have a public portfolio piece
that they can reference plus they would get to learn all about the open
source tools and meet folks. Having worked on an open source project is
impressive, more so than designing a site or app that never launched.

Even small examples that are like mini apps could get a face lift. The docs
could even get help with graphics that explain how stuff works. People that
do data visualization could have a productive time working with ML as well.

Having more creatives in the open source community, not just MXNet, would
be great for diversity.

Cheers,
Aaron


On Wed, Mar 6, 2019, 07:10 Isabel Drost-Fromm  wrote:

>
>
> Am 2. März 2019 15:13:23 MEZ schrieb Carin Meier :
> >I wanted to kickoff a discussion about community building. There was an
> >excellent blog post from the Apache Beam Community on this
> >https://blogs.apache.org/comdev/entry/an-approach-to-community-building
>
> Needless to say I really love that blog post.
>
> Other than that there are a couple of question you might ask yourself as a
> community:
>
> How easy is it to patriciate as an outsider, how much communication is
> happening outside of dev@?
>
> How explicit are you about how inclusive you want to be? See also
> https://youtu.be/LgB1s3buccI
>
> How explicit are you about where you need help (including and beyond
> coding)?
>
> How explicit are you with downstream users that some of the inner workings
> of Apache projects are build around a scratch your own itch casual
> contributions that ideally should be rewarded the same way as full-time
> contributions (
> https://blogs.apache.org/foundation/entry/success-at-apache-for-love )?
>
> I think you want to enable as many ppl as possible - typically only ten
> percent of your users turn into contributor, of those only ten percent
> trend to be repeat contributors... at least in my experience.
>
> Just some ideas,
> Isabel
> --
> Diese Nachricht wurde von meinem Android-Gerät mit K-9 Mail gesendet.
>


Re: Request for invitation to slack channel

2019-02-28 Thread Aaron Markham
I sent you an invite.

On Thu, Feb 28, 2019 at 7:11 AM Bazan, Ivy  wrote:
>
> I’d like to request an invitation to the #mxnet Slack channel.


direction for documentation across various APIs that share common doc source

2019-02-25 Thread Aaron Markham
Hi everyone,
A recent issue and pending PR has brought a thorny docs situation to
my attention again and I'd like to hear from the community on how to
proceed.
We currently get some of the docs for the Python API pulled out of .cc
files. Other APIs also get docs from there, or pull the Python docs to
autogenerate their docs. This presents some problems:
1. (Some of) The code examples provided don't run when you copy and
paste them. [1]
2. The code examples that show up in other APIs won't work as the code
is Python and for (many/complicated) statements the syntax can be
wrong.

When I try out something new and go for the hello world example or
browse around I do expect the docs' code examples to work. If they
don't, well, that's a bad sign and I move on to another project. I'd
like for new users to have a great experience no matter what language
they use.

One fix is to go ahead a be "Python 1st" and make sure the code
executes. This route is proposed in a PR for some NDArray operators.
[2] As I mention in the PR comments, this has the drawback of being
very specific to Python and the psuedo-code, for what its worth,
showing up in Scala docs (for example) will be much more obviously out
of place. If I were an Scala person, I'd probably find this
irritating. The same goes for R.

So... what should we do? Here are some ideas:
a) I thought about providing different examples in the .cc code, one
for each language and then making sure those are parsed out properly
when the APIs are generating their docs. I'm not sure how feasible
this is.
b) I thought that it would be nice if each operator had a wrapper for
each language API, and this is where the example payload resides.
Maybe docstrings go here too or the common docstrings just bubble up
from the cc file. The benefit is that changes for a specific language
remain in those packages and don't touch the shared core files.
c) Another route is to keep the examples in the .cc files pseudo-code,
but then also make sure each language has real examples in their docs.
Then, any code block that's in the docs now that won't execute should
be changed to a preformatted text block so people don't confuse it
with functional code.

I really don't like any of these options as they each sound like ton
of work and difficult to maintain. Are there any projects that solve
this problem in some elegant and efficient way?

Cheers,
Aaron

[1] https://github.com/apache/incubator-mxnet/issues/14232
[2] https://github.com/apache/incubator-mxnet/pull/14243


Re: [RESTARTING][VOTE] Release Apache MXNet (incubating) version 1.4.0.rc2

2019-02-13 Thread Aaron Markham
Sheng, thanks for being so proactive, but adding license headers to
the markdown files in #14142 breaks the website as I warned. I caught
it before it went live.
I've disabled website publishing until this situation is resolved.


On Wed, Feb 13, 2019 at 10:59 AM Sheng Zha  wrote:
>
> Update: All license issues mentioned in the general vote from Luciano (pom
> files, docker files, docs) have been fixed on master [1][2].
>
> Let me know if there's more to address.
>
> -sz
>
> [1] https://github.com/apache/incubator-mxnet/pull/14138
> [2] https://github.com/apache/incubator-mxnet/pull/14142
>
> On Wed, Feb 13, 2019 at 7:54 AM Michael Wall  wrote:
>
> > So is the plan option 3?  I have seen tickets fixing licenses, so good work
> > there.  When a vote is started on dev@mxnet.a.o, include wording about not
> > waiting the full 72 hours since this is just updating licensing.  Get as
> > many +1 votes as you can on both the release and not waiting then move on
> > to IPMC.  The vote on general@incubator.a.o should still stay open 72
> > hours.  I will look at it as soon as it is posted, but maybe reach out to
> > the other mentors directly asking for their help to review as soon as it is
> > out.  The goal is to have the 3 or more +1 votes and more positive then
> > negative as soon as the 72 hours hits.
> >
> > Mike
> >
> > On Wed, Feb 13, 2019 at 2:44 AM Justin Mclean 
> > wrote:
> >
> > > forgot to CC dev
> > >
> > > > Begin forwarded message:
> > > >
> > > > From: Justin Mclean 
> > > > Subject: Re: [RESTARTING][VOTE] Release Apache MXNet (incubating)
> > > version 1.4.0.rc2
> > > > Date: 13 February 2019 at 6:43:48 pm AEDT
> > > > To: Michael Wall 
> > > >
> > > > Hi,
> > > >
> > > >> Option 1:
> > > >> Do nothing.  I don't know how a RESTARTED vote works.
> > > >
> > > > I don’t believe there is such a concept.
> > > >
> > > >> Option 2:
> > > >> Start another vote thread on general@incubator.a.o pointing to the
> > > original vote thread on dev@mxnet.a.o and the canceled vote thread.
> > > >
> > > > It may end up with the same outcome.
> > > >
> > > >> Option 3:
> > > >> 1 - Fix the header issues.
> > > > 
> > > >> 3 - Start a vote thread on general@incubator.a.o pointing to the new
> > > vote thread from step 2.  Will likely need to be open 72 hours.
> > > >
> > > > Just be aware it can take longer, sometime much longer, to get the 3 +1
> > > IPMC votes.
> > > >
> > > >> Tough position to be in with Horovod being released.
> > > >
> > > > Which show the risk of tying in your release cycle with a non Apache
> > > product. IMO you need to be independent of 3rd party releases and not
> > tied
> > > to their milestones. If they wanted to include a particular unreleased
> > > version of ASF software, you should started the release a long time ahead
> > > of time just in case problems were encountered issues.This probably
> > > wouldn't be an issue if you made more frequent releases, it’s easier to
> > > check compliance with frequent releases so the 3rd party could just take
> > > the last good release and go with that.
> > > >
> > > > Thanks,
> > > > Justin
> > >
> > >
> >


Re: [RESTARTING][VOTE] Release Apache MXNet (incubating) version 1.4.0.rc2

2019-02-12 Thread Aaron Markham
d in ASF policy.  But
> rushing a vote does have risks, such as less testing on the code being
> released.
>
> To make this more confusing, the VOTE thread is showing up on both
> dev@mxnet.a.o and general@incubator.a.o.  There is an additional +1 vote on
> the dev@mxnet.a.o list that doesn't show up on the general@incubator, but
> this too is non binding best I can tell.
>
> Tough position to be in with Horovod being released.  Nothing in ASF policy
> makes allowances for such an event that I could find.  Perhaps we should
> ask for more clarification on general@incubator.a.o to get more thoughts
> from the IPMC.
>
> Mike
>
> On Tue, Feb 12, 2019 at 5:53 PM Qing Lan  wrote:
>
> > Hi Michael,
> >
> > Could you please guide how to proceed with this? Given that we have a
> > possibility of announcing MXNet support in Horovod with their next release
> > and this would help MXNet increase our visibility.
> >
> > Thanks,
> > Qing
> >
> > On 2/12/19, 2:16 PM, "Michael Wall"  wrote:
> >
> > Team,
> >
> > Here is my read on the situation.  The vote has been canceled.
> > Justin's
> > point was that a -1 doesn't mean you must cancel a vote for the
> > reasons he
> > outlined.  But here the vote needs to be restarted and the issue
> > Luciano
> > found needs to be addressed.
> >
> > That issue is that there are files in MXNet source tree that do not
> > have
> > the required licenses headers,
> > http://www.apache.org/legal/release-policy.html#license-headers.  For
> > example, the top level README.md is missing the header
> >
> > https://raw.githubusercontent.com/apache/incubator-mxnet/master/README.md.
> > Excluding 3rd party files from the RAT check is fine, but not files
> > originating from the MXNet repo.
> >
> > It would be good to know exactly how Luciano ran the RAT check, cc'd.
> > Here
> > is a link to the thread
> >
> > https://lists.apache.org/thread.html/51e9ab05edae2089c74a253000a92d5aa5c6406f54e5bd0a0b3c3879@%3Cgeneral.incubator.apache.org%3E
> > .
> >
> > Justin's other point, aIso cc'd, was that the vote with the podling
> > doesn't
> > have to take 72 hours before going to the incubator list.
> >
> > I realize this is not what everyone is pushing for, so interested in
> > other's thoughts.  Especially other mentors.
> >
> > Mike
> >
> > On Tue, Feb 12, 2019 at 4:47 PM Aaron Markham <
> > aaron.s.mark...@gmail.com>
> > wrote:
> >
> > > +1
> > > I disagree about 3rd party considerations. This is an ecosystem
> > after all.
> > > The distributed training story is quite nice with Horovod. Given my
> > > interaction with tensorflow with  Horovod and dynamic training with
> > MXNet
> > > and the kvstore, this new route is, IMO, easier to setup and manage.
> > > I see the benefit for getting it out there sooner than later, and
> > market
> > > timings are important to the project and adoption. If Uber's
> > announcement
> > > drives traffic to MXNet, but then people can't set it up with a
> > stable
> > > release package, there's a lost opportunity for growing the
> > community. Why
> > > miss the opportunity for a RAT license?
> > >
> > > On Tue, Feb 12, 2019, 13:14 Dave Fisher 
> > wrote:
> > >
> > > > Hi -
> > > >
> > > > Third party vendor considerations do not matter. Are you voting +1
> > with
> > > > your Apache hat on or your Amazon hat?
> > > >
> > > > Regards,
> > > > Dave
> > > >
> > > > > On Feb 11, 2019, at 10:16 PM, Lin Yuan 
> > wrote:
> > > > >
> > > > > +1 binding
> > > > > Horovod is going to release it's 0.16.0 in the coming week with
> > MXNet
> > > > > integration. We need to release 1.4.0 which includes all the
> > > dependencies
> > > > > for Horovod integration.
> > > > >
> > > > > Best,
> > > > >
> > > > > Lin
> > > > >
> > > > > On Mon, Feb 11, 2019 at 9:30 PM Steffen Rochel <
> > > steffenroc...@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> Dear co

Re: [RESTARTING][VOTE] Release Apache MXNet (incubating) version 1.4.0.rc2

2019-02-12 Thread Aaron Markham
+1
I disagree about 3rd party considerations. This is an ecosystem after all.
The distributed training story is quite nice with Horovod. Given my
interaction with tensorflow with  Horovod and dynamic training with MXNet
and the kvstore, this new route is, IMO, easier to setup and manage.
I see the benefit for getting it out there sooner than later, and market
timings are important to the project and adoption. If Uber's announcement
drives traffic to MXNet, but then people can't set it up with a stable
release package, there's a lost opportunity for growing the community. Why
miss the opportunity for a RAT license?

On Tue, Feb 12, 2019, 13:14 Dave Fisher  wrote:

> Hi -
>
> Third party vendor considerations do not matter. Are you voting +1 with
> your Apache hat on or your Amazon hat?
>
> Regards,
> Dave
>
> > On Feb 11, 2019, at 10:16 PM, Lin Yuan  wrote:
> >
> > +1 binding
> > Horovod is going to release it's 0.16.0 in the coming week with MXNet
> > integration. We need to release 1.4.0 which includes all the dependencies
> > for Horovod integration.
> >
> > Best,
> >
> > Lin
> >
> > On Mon, Feb 11, 2019 at 9:30 PM Steffen Rochel 
> > wrote:
> >
> >> Dear community -
> >> based on Justin's and community feedback I'm suggesting to restart the
> >> vote.
> >> Current status:
> >> binding votes:
> >> +1: 2 votes (Henri, Jason)
> >> -1:  1 vote (Luciano)
> >>
> >> non-binding:
> >> +1: 1 vote (Kellen)
> >>
> >> The community is investigating feedback from Luciano that the exclusion
> >> file is to broad and potentially missing files which can and must have
> >> apache license headers not to be checked.
> >>
> >> Regards,
> >> Steffen
> >>
> >>
> >>
> >>
> >> On Mon, Feb 11, 2019 at 10:08 AM Hagay Lupesko 
> wrote:
> >>
> >>> Based on Justin's feedback, can we resume the vote instead of
> cancelling
> >>> it?
> >>>
> >>> On Mon, Feb 11, 2019 at 12:02 AM Justin Mclean <
> jus...@classsoftware.com
> >>>
> >>> wrote:
> >>>
>  Hi,
> 
>  In future don’t be so hasty to cancel a release vote, people mind can
> >> be
>  changed and a -1 is not a veto on a release.
> 
>  Thanks,
>  Justin
> 
> 
>  -
>  To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
>  For additional commands, e-mail: general-h...@incubator.apache.org
> 
> 
> >>>
> >>
>
>


Re: Benchmarking MXNet with different compilers and different OpenMP implementations (results)

2019-02-12 Thread Aaron Markham
This is really great research. I've often wondered what the difference
really is, and why it has to be so complicated. It seems the answer is
there isn't much difference and it shouldn't be as complex.
As for your next steps, would you propose that cmake be brought up to
parity? It seems strange that it causes slowness and if so, it shouldn't be
recommended for now.
Also, testing for windows compliers might be quite important as install
stats suggest a significant portion of windows users. Wouldn't this nudge
the decision of what to use as a rule going forward?
I ran into this submodule openmp issue on windows myself. How does that get
fixed? Do we have to repackage all of the submodules to make sure they use
the recommended implementation or they use what the system expects?

Cheers,
Aaron

On Tue, Feb 12, 2019, 04:37 Anton Chernov  wrote:

> Dear MXNet community,
>
> Due to multiple problems related to OpenMP and stale proposed change [1] we
> have been working on gathering performance data on the impact of using
> different OpenMP implementations with MXNet (great thanks to Stanislav
> Tsukrov for the hard work). The results can be found here [2].
>
> As a short summary of the investigation: The difference between different
> compilers is insignificant. Native OpenMP implementations (more or less
> recent) perform equally (<5% difference). See more details in the document.
>
> Please review the document and share your thoughts on the topic.
>
> Thanks!
>
> Best
> Anton
>
> [1]
>
> https://lists.apache.org/thread.html/4827f0f742b6e7e070da350ea81226d059401527f3072ce8b33c1fdf@
> 
> [2] https://cwiki.apache.org/confluence/x/2wclBg
>


Re: website updates & redesign

2019-02-04 Thread Aaron Markham
Thanks Marco. Any tutorial that is executable should have the minimum
version indicated in its prerequisites section. Even so, we're not
testing tutorials against previous versions so while the info would be
stated we'd only really be testing against master, AFAIK.
I'm inclined to creates guides on how to generate legacy documentation
for each API, and remove the complexity of versions from the CI
pipeline. One-off generated API docs of old versions could be hosted,
but not part of the publishing and CI pipeline. This would be more
like PyTorch. https://pytorch.org/docs/versions.html We have to do a
bit better than that since we have so many language bindings with
varying degrees of support across versions. And we should make sure
that search works across versions and is clearly labelled, so you know
what you're looking at.

On Mon, Feb 4, 2019 at 4:30 PM Marco de Abreu  wrote:
>
> Great design!
>
> The different versioning is a good point, thanks for bringing it up. Could
> you elaborate on how we will handle the situation of different versions of
> mxnet in future (e.g. examples that require a minimum version)? I assume we
> will revamp the website build process as part of these efforts, right?
>
> Best regards
> Marco
>
> Am Di., 5. Feb. 2019, 01:22 hat Aaron Markham 
> geschrieben:
>
> > Hello MXNet folks!
> >
> > As I mentioned last week, I'm soliciting feedback on a website
> > redesign. Some of you may have seen the beta site that Mu has been
> > working on, which is a big step in the right direction for the
> > automatically generated API docs. I've posted a wiki article [1] that
> > proposes how to flesh out the new site with an information
> > architecture, user flows, and wireframes useful for a design
> > treatment. I look forward to your feedback - if not here or on the
> > wiki, let's get a discussion going in Slack. Just ping me there.
> >
> > As a side note and follow up to fixing the redirects on the website, I
> > needed to add an artifacts flag in the doc's settings.ini file [2].
> > This setting is triggered in mxdoc.py and lets you take files needed
> > for the docs build from master (or your branch) and apply them to old
> > version branches that you're building for the website. This also
> > transitions some of logic from the post-processing versions update
> > step during web publishing.
> >
> > [1] Website redesign information architecture and wireframe:
> > https://cwiki.apache.org/confluence/x/vAMlBg
> > [2] Website publish updates:
> > https://github.com/apache/incubator-mxnet/pull/14015
> > Cheers,
> > Aaron
> >


website updates & redesign

2019-02-04 Thread Aaron Markham
Hello MXNet folks!

As I mentioned last week, I'm soliciting feedback on a website
redesign. Some of you may have seen the beta site that Mu has been
working on, which is a big step in the right direction for the
automatically generated API docs. I've posted a wiki article [1] that
proposes how to flesh out the new site with an information
architecture, user flows, and wireframes useful for a design
treatment. I look forward to your feedback - if not here or on the
wiki, let's get a discussion going in Slack. Just ping me there.

As a side note and follow up to fixing the redirects on the website, I
needed to add an artifacts flag in the doc's settings.ini file [2].
This setting is triggered in mxdoc.py and lets you take files needed
for the docs build from master (or your branch) and apply them to old
version branches that you're building for the website. This also
transitions some of logic from the post-processing versions update
step during web publishing.

[1] Website redesign information architecture and wireframe:
https://cwiki.apache.org/confluence/x/vAMlBg
[2] Website publish updates:
https://github.com/apache/incubator-mxnet/pull/14015
Cheers,
Aaron


Re: [VOTE] Release Apache MXNet (incubating) version 1.4.0.rc2

2019-01-31 Thread Aaron Markham
+1 I built the full website including the branch and it was fine.

On Wed, Jan 30, 2019, 22:39 kellen sunderland  +0
>
> Overall release looks good.  Probably something I'm doing wrong, but so far
> not able to validate the .asc.  I'm getting "Can't check signature: No
> public key".  I've added the keys from GitHub and the release folder, and
> also added your public key "40C9346904DFCE37" from the MIT key server
> Steffen.  Is there another key I'm missing?
>
> 1. sha512 look good.
> 2. Compile from source successfully
> 3. TensorRT build succeeds and runs inference for demo models
> 4. License, notice and disclaimer exist.
>
> -Kellen
>
> On Wed, Jan 30, 2019 at 8:58 PM Steffen Rochel 
> wrote:
>
> > Dear MXNet community -
> > we currently have three +1 votes, one binding.
> > As the vote did not reach the necessary number of binding votes I'm
> > extending voting.
> >
> > I'm calling on all PMC member, please test and vote.
> >
> > Regards,
> > Steffen
> >
> > On Wed, Jan 30, 2019 at 6:43 PM Aston Zhang 
> wrote:
> >
> > > +1
> > >
> > > Tested with the Dive into Deep Learning book.
> > >
> > > On Wed, Jan 30, 2019 at 1:25 PM Steffen Rochel <
> steffenroc...@gmail.com>
> > > wrote:
> > >
> > > > Thanks Carin and Yuxi.
> > > >
> > > > Committers and PMC members - please test and send your vote to
> release
> > > > Apache MXNet (incubating) v1.4.0.
> > > >
> > > > Regards,
> > > > Steffen
> > > >
> > > > On Wed, Jan 30, 2019 at 10:55 AM Yuxi Hu 
> wrote:
> > > >
> > > > > +1
> > > > >
> > > > > verified the training throughput for resnet50_v1 looks normal
> > compared
> > > to
> > > > > 1.3.1 release
> > > > >
> > > > > On Tue, Jan 29, 2019 at 3:36 PM Carin Meier 
> > > > wrote:
> > > > >
> > > > > > +1 - checked out from the release tag and built and tested
> > > > Scala/Clojure
> > > > > > package.
> > > > > >
> > > > > > On Sat, Jan 26, 2019 at 8:53 PM Steffen Rochel <
> > > > steffenroc...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Dear MXNet community,
> > > > > > >
> > > > > > > This is the vote to release Apache MXNet (incubating) version
> > > v1.4.0.
> > > > > > > Voting will
> > > > > > > start today, Saturday January 26th 6pm PST and will close on
> > > > Wednesday,
> > > > > > > January 30th 7pm PST.
> > > > > > >
> > > > > > > Link to release notes:
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/MXNET/Apache+MXNet+%28incubating%29+
> > > > > > > 1.4.0+Release+Notes
> > > > > > >
> > > > > > > Link to release candidate:
> > > > > > > https://github.com/apache/incubator-mxnet/releases/tag/
> > > > > > > <
> > https://github.com/apache/incubator-mxnet/releases/tag/1.4.0.rc2
> > > > > > >1.4.0.rc
> > > > > > > <
> > https://dist.apache.org/repos/dist/dev/incubator/mxnet/1.4.0.rc1
> > > >2
> > > > > > >
> > > > > > > Link to source and signatures on apache dist server:
> > > > > > >
> https://dist.apache.org/repos/dist/dev/incubator/mxnet/1.4.0.rc2
> > > > > > >
> > > > > > >
> > > > > > > Please remember to TEST first before voting accordingly:
> > > > > > > +1 = approve
> > > > > > > +0 = no opinion
> > > > > > > -1 = disapprove (provide reason)
> > > > > > >
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Steffen
> > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Yuxi(Darren) Hu, Ph.D.
> > > > > Software Development Engineer
> > > > > Amazon Web Services
> > > > >
> > > >
> > >
> >
>


Re: Horovod-MXNet Integration

2019-01-30 Thread Aaron Markham
Congrats on the Horovod integration everyone. That's really great to hear.

On Wed, Jan 30, 2019 at 10:08 AM Lin Yuan  wrote:
>
> Hi Yuan,
>
> Thanks for your interest. We have just supported MXNet in Horovod and are
> working on performance tuning and adding more examples. We are definitely
> interested in further extending it's support with Kubeflow.
>
> Let's set up some time to have a more detailed discussion.
>
> Best,
>
> Lin
>
> On Wed, Jan 30, 2019 at 7:42 AM Yuan Tang  wrote:
>
> > Hi,
> >
> > It's great to see MXNet-Horovod integration got merged:
> > https://github.com/uber/horovod/pull/542
> >
> > Is there any future plan for this? I've been working on Kubeflow's
> > MPI-Operator (https://github.com/kubeflow/mpi-operator) lately and it
> > would
> > be interesting to see an example of using Horovod + MXNet + Kubeflow using
> > MPI Operator. Feel free to reach out (@terrytangyuan
> > ) if you encounter any issues.
> >
> > Best,
> > Yuan
> >
> >
> > On Fri, Nov 2, 2018 at 6:51 PM Lin Yuan  wrote:
> >
> > > Hi Mu,
> > >
> > > Darren (@yuxihu ) and I have been working on
> > > releasing MXNet-Horovod integration in production. We have made some
> > > changes on both MXNet and Horovod sides. The changes on MXNet side have
> > > mostly been merged and we are working to merge code to horovod repo. We
> > > will send a design doc to you for review again next week.
> > >
> > > Thanks for your feedback,
> > >
> > > Lin
> > >
> > > On Wed, Oct 31, 2018 at 12:03 PM Mu Li  wrote:
> > >
> > > > Thanks for your contribution, Carl.
> > > >
> > > > I remember I left a comment on the proposal, but today I found it was
> > > > disappeared. My suggestion is trying best to not change the existing
> > API.
> > > > The reason is that we need to change all trainers on the frontend that
> > > uses
> > > > the existing kvstore APIs, which may cause confusion to users.
> > > >
> > > > The current proposal wants add the following 4 APIs into kvstore:
> > > >
> > > >
> > > >-
> > > >
> > > >kv.pushpull
> > > >-
> > > >
> > > >kv.broadcast
> > > >-
> > > >
> > > >kv.local_rank
> > > >-
> > > >
> > > >kv.num_local_workers
> > > >
> > > >
> > > > Pushpull can be done with a sequential push and pull, you can do
> > nothing
> > > in
> > > > push and put all workloads into pushpull. Broadcast can be implemented
> > by
> > > > pull.
> > > >
> > > > What's local workers? GPUs in the single machine? If so, we can query
> > it
> > > > directly.
> > > >
> > > >
> > > > On Fri, Sep 14, 2018 at 4:46 PM Carl Yang  wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Currently, MXNet distributed can only be done using parameter server.
> > > > > Horovod is an open-source distributed training framework that has
> > > > > shown 2x speedup compared to TensorFlow using Parameter Server. We
> > > > > propose to add Horovod support to MXNet. This will help our users
> > > > > achieve goal of linear scalability to 256 GPUs and beyond. Design
> > > > > proposal on cwiki:
> > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/MXNET/Horovod-MXNet+Integration
> > > > >
> > > > > Please feel free to let me know if you have any suggestions or
> > > feedback.
> > > > >
> > > > > Regards,
> > > > > Carl
> > > > >
> > > >
> > >
> >


website, docs & related features updates

2019-01-28 Thread Aaron Markham
Hello dev folks! I thought I'd let you all know some of the recent
updates related to the website, docs & related features.

MXNet CI and website build pipeline

We had few blips last week from where GitHub was unreachable. The red
block in the trend graph [1] is from the previous week’s subprocess
issue. I see that the general MKL-DNN and subprocess problem is still
not resolved [2]. Please let me know if there's something that should
be clarified in instructions because the error messaging is really not
great, and someone else running into this issue might waste a lot of
time troubleshooting it.


MXNet Website

* Error handling PR merged [3] – the redirects have been updated and I
reimplemented the error pages. They now provide a real 404 and I
imagine we’ll start getting broken link reports now. (So, I’ll be
looking at that this week, plus I will track/scope out and possibly
implement a versions dropdown or cross-versions search for these
pages.)
* C++ API docs improvement [4]– some functions and features were not
getting documented because op.h is only built when you build MXNet
with the CPP flag. This has been added and you will now be able to see
the docs for this. Note that if you run a docs build, you’ll see a ton
of warnings [5] coming from this file’s Doxygen processing. This needs
some attention. Maybe someone that knows this part of the codebase can
take a look?


MXNet Beta Website

I’m scoping out some of the remaining work to accelerate the
production launch for the beta website [6]. I’ll be looking at this
more this week and would appreciate feedback and suggestions.


MXNet Features

Looks like the lipreading example [7] from @seujung is ready. If you
haven't tried it out yet, take a look. I put the preprocessed training
set on S3 to save you time. (aws s3 sync s3://mxnet-public/lipnet/). I
think a blog post about it would be a nice follow-up to having this
merged. What would be really cool is another example using soccer
footage (ref and/or player exchanges) or other such things with
amusing or interesting outputs. Making it turnkey for people writing
accessibility apps would be great.


[1] Trend graph for website publishing job:
http://jenkins.mxnet-ci.amazon-ml.com/job/restricted-website-build/buildTimeTrend
[2] MKL-DNN subprocess issue:
https://github.com/apache/incubator-mxnet/issues/13875
[3] Error handling PR: https://github.com/apache/incubator-mxnet/pull/13963
[4] C++ API docs improvement:
https://github.com/apache/incubator-mxnet/pull/13983
[5] Comment about op.h errors:
https://github.com/apache/incubator-mxnet/issues/13920#issuecomment-455653127
[6] Beta website: http://beta.mxnet.io
[7] Lipreading example: https://github.com/apache/incubator-mxnet/pull/13647


Re: Taxonomy on our cwiki

2019-01-18 Thread Aaron Markham
When the home page was renamed to apache-mxnet from just MXNet it said the
page no longer existed, so we should verify anything in the website and
update accordingly.

On Fri, Jan 18, 2019, 17:20 Haibin Lin  +1
>
> Will there be broken links? I thought confluence will show "page is now
> moved to https://xxx.html; to redirect users, when this kind of reorg
> happens.
>
> Best,
> Haibin
>
> On Fri, Jan 18, 2019 at 4:50 PM Aaron Markham 
> wrote:
>
> > +1 but note that this is probably going to create a bunch of broken links
> > on the MXNet website and maybe elsewhere. Should make time to deal with
> > that in this process.
> >
> > On Fri, Jan 18, 2019, 12:43 Carin Meier  >
> > > +1 Great idea
> > >
> > > On Fri, Jan 18, 2019 at 2:38 PM Sheng Zha  wrote:
> > >
> > > > Hi MXNet,
> > > >
> > > > Given that currently cwiki is the only place other than mxnet website
> > for
> > > > mxnet-related documentation, I'd like to request your attention to
> the
> > > > (slightly disorganized) cwiki page of MXNet. The top level folders
> (and
> > > > their contents) currently looks like this:
> > > > - Design Proposals* (bag of proposals, not in order)
> > > > - Development* (mixture of guides, roadmaps, processes)
> > > > - Release Process (release notes)
> > > > - Website (guides and proposals)
> > > > - MXNet Clojure (call for contribution, guides)
> > > > - MXNet Keras Integration (design)
> > > > - MXNet-ONNX Integration (design, dev status)
> > > > - MXNet R Package (guide, backlog)
> > > > - MXNet-Scala (design, dev status, guide)
> > > > - Content Formatting Templates (not a folder but link to two docs)
> > > > - How-to articles (1 guide)
> > > > - Community (guide on apache-related processes)
> > > > - Data IO (designs)
> > > > - Continuous Integration (guides, designs)
> > > > - Meetups and Hangouts (events)
> > > >
> > > > And here are two good examples from successful Apache projects:
> > > > - Apache Flink: an **audience-oriented** structure [1]
> > > >   Users (Presentations and How-to)
> > > >   Contributors (Dev processes and How-to)
> > > >   Committers (Infra, Dev processes, Release processes, Releases)
> > > >   Roadmaps and Feature Designs (archive)
> > > > - Apache OpenNLP: a **content-oriented** structure [2]
> > > >   Guides
> > > >   External Resources
> > > >   Proposals
> > > >   Releasing
> > > >
> > > > Clean organization helps content discovery and saves time on locating
> > > > useful content. Given that we have good amount of content on the wiki
> > > page,
> > > > I suggest that we decide on a cleaner taxonomy, re-organize contents
> > > > accordingly, and add future contents accordingly. To provide a
> starting
> > > > point for the discussion, I suggest:
> > > > - Given the state we are in, start with content-oriented
> organization,
> > > use
> > > > these top-level categories: Guides (including processes and how-tos),
> > > > Development (including designs, proposals, notes, roadmaps),
> Community
> > > > (including events, activities, external resources and contents)
> > > > - If people strongly prefer audience-oriented structure, later we can
> > > adopt
> > > > a structure similar to Flink's.
> > > >
> > > > Feel free to share your thoughts and preferences here. Thanks.
> > > >
> > > > -sz
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Apache+Flink+Homehttps://cwiki.apache.org/confluence/display/FLINK/Apache+Flink+Home
> > > > [2] https://cwiki.apache.org/confluence/display/OPENNLP/Index
> > > >
> > >
> >
>


Re: Taxonomy on our cwiki

2019-01-18 Thread Aaron Markham
+1 but note that this is probably going to create a bunch of broken links
on the MXNet website and maybe elsewhere. Should make time to deal with
that in this process.

On Fri, Jan 18, 2019, 12:43 Carin Meier  +1 Great idea
>
> On Fri, Jan 18, 2019 at 2:38 PM Sheng Zha  wrote:
>
> > Hi MXNet,
> >
> > Given that currently cwiki is the only place other than mxnet website for
> > mxnet-related documentation, I'd like to request your attention to the
> > (slightly disorganized) cwiki page of MXNet. The top level folders (and
> > their contents) currently looks like this:
> > - Design Proposals* (bag of proposals, not in order)
> > - Development* (mixture of guides, roadmaps, processes)
> > - Release Process (release notes)
> > - Website (guides and proposals)
> > - MXNet Clojure (call for contribution, guides)
> > - MXNet Keras Integration (design)
> > - MXNet-ONNX Integration (design, dev status)
> > - MXNet R Package (guide, backlog)
> > - MXNet-Scala (design, dev status, guide)
> > - Content Formatting Templates (not a folder but link to two docs)
> > - How-to articles (1 guide)
> > - Community (guide on apache-related processes)
> > - Data IO (designs)
> > - Continuous Integration (guides, designs)
> > - Meetups and Hangouts (events)
> >
> > And here are two good examples from successful Apache projects:
> > - Apache Flink: an **audience-oriented** structure [1]
> >   Users (Presentations and How-to)
> >   Contributors (Dev processes and How-to)
> >   Committers (Infra, Dev processes, Release processes, Releases)
> >   Roadmaps and Feature Designs (archive)
> > - Apache OpenNLP: a **content-oriented** structure [2]
> >   Guides
> >   External Resources
> >   Proposals
> >   Releasing
> >
> > Clean organization helps content discovery and saves time on locating
> > useful content. Given that we have good amount of content on the wiki
> page,
> > I suggest that we decide on a cleaner taxonomy, re-organize contents
> > accordingly, and add future contents accordingly. To provide a starting
> > point for the discussion, I suggest:
> > - Given the state we are in, start with content-oriented organization,
> use
> > these top-level categories: Guides (including processes and how-tos),
> > Development (including designs, proposals, notes, roadmaps), Community
> > (including events, activities, external resources and contents)
> > - If people strongly prefer audience-oriented structure, later we can
> adopt
> > a structure similar to Flink's.
> >
> > Feel free to share your thoughts and preferences here. Thanks.
> >
> > -sz
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Apache+Flink+Homehttps://cwiki.apache.org/confluence/display/FLINK/Apache+Flink+Home
> > [2] https://cwiki.apache.org/confluence/display/OPENNLP/Index
> >
>


CI reporting & diagnostics improvements

2019-01-17 Thread Aaron Markham
I've been trying to diagnose issues with some of the CI pipelines and one
blind spot in the reporting is in the stages - we'll have timing info on
the first couple of parts and maybe the last part, but the lion's share is
in one big part in the middle. It'll say 45 minutes, or jump to an hour,
and I can't tell why because there's not enough granularity in the report.
Is that something we can improve?
For example, this one jumped to almost 2 hours from a normal 45 minute run:
http://jenkins.mxnet-ci.amazon-ml.com/job/restricted-website-build/986/timings/

I'd also like to see some data dumps like memory usage and disk space and
whatever else diagnostics we can get and each stage.

Cheers,
Aaron


Re: joining slack channel

2019-01-05 Thread Aaron Markham
On the left side, click Channels, then you will see a search option. Look
for mxnet there.

On Fri, Jan 4, 2019 at 6:45 PM Fehervari, Istvan
 wrote:

> Thanks for the invite. I am new to Slack so this might be a noob question
> but I only see the #general and #welcome channels in ASF. How do I join the
> #mxnet channel?
>
> Thanks a lot!
> Istvan
>
> On 1/4/19, 5:44 AM, "Aaron Markham"  wrote:
>
> I sent you an invite.
> Cheers,
> Aaron
>
> On Fri, Jan 4, 2019 at 1:12 PM István Fehérvári 
> wrote:
>
> > Please add me to the slack channel.
> >
> > Thank you,
> > Istvan
> >
>
>
>


Re: joining slack channel

2019-01-04 Thread Aaron Markham
I sent you an invite.
Cheers,
Aaron

On Fri, Jan 4, 2019 at 1:12 PM István Fehérvári  wrote:

> Please add me to the slack channel.
>
> Thank you,
> Istvan
>


Re: Running CI for doc changes

2019-01-04 Thread Aaron Markham
Hi Istvan,
You can make a page here:
https://cwiki.apache.org/confluence/display/MXNET/Website+Update+Proposals
If you don't have an account yet you can sign up, and then we can check on
edit access.
Also if you're not on slack yet, please join and we can also chat there.

With regard to the files, for .md generating code, are you referring to the
tutorials?
While probably overlooking something, I think the first pass on this would
be to check for changes to /docs/* then limit the CI process to the docs
pipeline. There's a tutorial validation pipeline that could be another
check - if something in /docs/tutorials/* gets updated then trigger that
pipeline...

I tried this API call to get a list of commits:
https://api.github.com/repos/apache/incubator-mxnet/pulls/13769/commits
But I was rate limited. So unless there's another way, the solution will
require an API key and management of the API calls. This might exist
somewhere already since we have the label bot.

Then this could be tried to get a list of files...
https://stackoverflow.com/questions/424071/how-to-list-all-the-files-in-a-commit
Then grep that list...
Then trigger pipelines accordingly...

Cheers,
Aaron


On Fri, Jan 4, 2019 at 6:42 AM István Fehérvári  wrote:

> I sorta got a simple skipping logic working however there seems to be a
> problem I cannot solve. In order to figure out whether an effecting change
> has happened I need a list of all commits in the PR and all the files in
> it. Unfortunately, currentBuild.changeSets returns data only about the head
> commit of the PR since jenkins assembles the build that way. It is even
> noted in the log after the merge of the master branch (First time build.
> Skipping changelog.)
>
> Does anyone have any idea how to retrieve all the commits in the PR?
>
> Also I naively thought that .md files can be skipped for the non-website
> pipelines however I was told that some of those files are indeed used for
> generating code. So for an effective strategy we need to understand what
> can be skipped in what cases: Marco mentioned language changes (R, scala)
> which I can add, any other ideas about strategies? Also I would like to
> document the strategies before coding them into a wiki of some sort, but
> not sure where I am supposed to do that (write access?).
>
> Thanks, for the help!
>
> Best,
> Istvan
>
> On Thu, Jan 3, 2019 at 2:18 PM István Fehérvári  wrote:
>
> > Hello Marco,
> >
> > Your idea is very much in line how I was imaging it. I will try to come
> up
> > with a prototype around this idea then we can continue the discussion
> about
> > specifics.
> >
> > Best,
> > Istvan
> >
> > On Thu, Jan 3, 2019 at 11:43 AM Marco de Abreu 
> > wrote:
> >
> >> Hello István,
> >>
> >> thanks for your interest and for offering your assistance! This is
> >> definitely a great idea and I think it has been mentioned a few times,
> but
> >> nobody jumped on it yet. We would be very happy to assist you on these
> >> efforts.
> >>
> >> As far as I can tell, it would boil down to having some kind of custom
> >> groovy function that we could call within our Jenkinsfile pipelines.
> This
> >> function would then determine whether that specific job/node should run
> or
> >> not. This could have two granularity levels:
> >> 1. Run or skip whole Jenkins job
> >> 2. Run or skip Jenkins node
> >>
> >> To clarify the terminology: Each entry (e.g. windows-cpu) at [1]
> (sourced
> >> from [2]) is a Jenkins job. Each Jenkins job can contain multiple stages
> >> (irrelevant here) and each stage contains one or more nodes. One green
> >> circle here [3] (e.g. Python 3: CPU Win) represents a node.
> >>
> >> #1 would be your example use case. In case of a doc change, we would
> skip
> >> our unit test pipelines while the website pipeline would still be
> >> executed.
> >> #2 would be language specific changes, for example. Imagine a change to
> >> the
> >> Scala code. This would require all Scala and Clojure jobs to re-run, but
> >> there would be no need to run R, Julia or Python, for example.
> >>
> >> One thought how to do this would be to define a mapping that would
> contain
> >> the "watched" directories for a particular Jenkins job or node. Before a
> >> job or node is triggered, it could then evaluate the previously
> mentioned
> >> function to determine whether any file within that mapping has changed.
> >>
> >> If you have any further questions or would like to discuss a design
> >> proposal, please don't hesitate to reach out to us again :)
> >>
> >> Best regards,
> >> Marco
> >>
> >> [1]: http://jenkins.mxnet-ci.amazon-ml.com/job/mxnet-validation/
> >> [2]: https://github.com/apache/incubator-mxnet/tree/master/ci/jenkins
> >> [3]:
> >>
> >>
> http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fwindows-cpu/detail/master/150/
> >>
> >> On Thu, Jan 3, 2019 at 7:22 PM István Fehérvári 
> wrote:
> >>
> >> > Hello developers,
> >> >
> >> > I recently opened a PR about a very small 

Re: CI impaired

2018-11-21 Thread Aaron Markham
Marco, thanks for your hard work on this. I'm super excited about the new
Jenkins jobs. This is going to be very helpful and improve sanity for our
PRs and ourselves!

Cheers,
Aaron

On Wed, Nov 21, 2018, 05:37 Marco de Abreu
 Hello,
>
> the CI is now back up and running. Auto scaling is working as expected and
> it passed our load tests.
>
> Please excuse the caused inconveniences.
>
> Best regards,
> Marco
>
> On Wed, Nov 21, 2018 at 5:24 AM Marco de Abreu <
> marco.g.ab...@googlemail.com>
> wrote:
>
> > Hello,
> >
> > I'd like to let you know that our CI was impaired and down for the last
> > few hours. After getting the CI back up, I noticed that our auto scaling
> > broke due to a silent update of Jenkins which broke our
> upscale-detection.
> > Manual scaling is currently not possible and stopping the scaling won't
> > help either because there are currently no p3 instances available, which
> > means that all jobs will fail none the less. In a few hours, the auto
> > scaling will have recycled all slaves through the down-scale mechanism
> and
> > we will be out of capacity. This will lead to resource starvation and
> thus
> > timeouts.
> >
> > Your PRs will be properly registered by Jenkins, but please expect the
> > jobs to time out and thus fail your PRs.
> >
> > I will fix the auto scaling as soon as I'm awake again.
> >
> > Sorry for the caused inconveniences.
> >
> > Best regards,
> > Marco
> >
> >
> > P.S. Sorry for the brief email and my lack of further fixes, but it's
> > 5:30AM now and I've been working for 17 hours.
> >
>


Re: [VOTE] Release Apache MXNet (incubating) version 1.3.1.rc0

2018-11-18 Thread Aaron Markham
I thought about it some more, and I don't think these are blockers.

+1 non-binding

Here's why:
For 1) The website splits things out properly and that's the primary
use case. If there's a demand for it, the jars for the docs can be
build per version and split out as needed, and these can be provided
for download.

For 2) This is an issue when you use a more recent version of Scala
than what we have in our CI and Ubuntu dependencies. It really only
pops up if you want to build scala, and you have the latest scala from
brew or if you have specifically upgraded scala on Ubuntu (where it
seems to be lagging a bit behind the version mac is using by default).
The release notes can have an entry that informs people three
different ways to resolve the issue: how to turn off the scala docs
build in the docss/settings.ini file, or the scala version to use, or
the line to update in mxdoc.py to get around the issue. For the next
release, we could upgrade scala in the dependencies and provide a
little better cross-platform support, but either way it shouldn't
happen in v1.4 since it's already patched in master.

Cheers,
Aaron
On Sat, Nov 17, 2018 at 1:05 PM Aaron Markham  wrote:
>
> Website and docs for 1.3.x branch is working fine. However...
>
> Two things to note that could be blockers:
> 1) The Scala API going out with the JavaAPI bundled inside it:
> /api/scala/docs/index.html#org.apache.mxnet.javaapi.package - this is
> fixed in master with #13071.
> 2) `make docs` doesn't work when you checkout v1.3.x branch on its
> own. The scala docs step breaks the build. I tested a fix with one
> update from the same Java/Scala docs PR I put in 4 days ago:
> https://github.com/apache/incubator-mxnet/pull/13071/files#diff-c8ca1b4ea12114fbd742c76087bca0e4R121
> This appears to fix the issue. (Scala version differences create
> different artifacts which may cause that step to fail.) I can create a
> PR for just this one change. I wouldn't apply #13071 - I tried it and
> it failed with Java-related dependency errors. If we want to fix both,
> then, maybe we can tackle that and the other errors and warnings that
> were just introduced with the Java API too.
> On Fri, Nov 16, 2018 at 8:44 PM sandeep krishnamurthy
>  wrote:
> >
> > +1
> >
> > Build from source on Mac and Ubuntu 16.04.
> >
> > 1. Run GAN, DCGAN using Gluon on 1 GPU in Ubuntu and CPU Mac.
> > 2. Convert a model to fp16, save and load in Gluon, perform inference.
> >
> > Best,
> > Sandeep
> >
> > On Fri, Nov 16, 2018, 7:01 PM Indhu  >
> > > +1.
> > >
> > > Builds fine from source. Tested few CNN examples using Python binding.
> > > Everything looks fine.
> > >
> > > Thanks,
> > > Indu
> > >
> > >
> > > On Fri, Nov 16, 2018, 4:16 PM Roshani Nagmote  > > wrote:
> > >
> > > > +1
> > > > Installed the release package on Ubuntu with CUDA, CUDNN enabled and ran
> > > > train_cifar10.py example successfully.
> > > >
> > > > Thanks,
> > > > Roshani
> > > >
> > > > On Fri, Nov 16, 2018 at 5:21 AM Anton Chernov 
> > > wrote:
> > > >
> > > > > Thank you Carin, Steffen and Kellen for your votes.
> > > > >
> > > > > The results so far:
> > > > >
> > > > > * Binding *
> > > > >
> > > > > +1 votes:
> > > > >   - Carin
> > > > >
> > > > > * Non-Binding *
> > > > >
> > > > > +1 votes:
> > > > >   - Kellen
> > > > >   - Steffen
> > > > >
> > > > > So far, we've got only positive votes, but unfortunately not enough to
> > > > > conclude a result from the vote. I would like to remind everyone that
> > > we
> > > > > need at least 3 binding +1 votes. Therefore, the vote is extended 
> > > > > until
> > > > > Tuesday 20th of November 2018, 5pm CET (9am PT).
> > > > >
> > > > > I kindly ask the community to participate in voting for this patch
> > > > release.
> > > > >
> > > > > Best regards
> > > > > Anton
> > > > >
> > > > >
> > > > > пт, 16 нояб. 2018 г. в 9:10, kellen sunderland <
> > > > > kellen.sunderl...@gmail.com
> > > > > >:
> > > > >
> > > > > > Just tested with 1.3.0 and those tests were failing for that release
> > > as
> > > > > > well.  Given it's not a regression I'm +1 (non-binding).
> > &g

Re: [VOTE] Release Apache MXNet (incubating) version 1.3.1.rc0

2018-11-17 Thread Aaron Markham
Website and docs for 1.3.x branch is working fine. However...

Two things to note that could be blockers:
1) The Scala API going out with the JavaAPI bundled inside it:
/api/scala/docs/index.html#org.apache.mxnet.javaapi.package - this is
fixed in master with #13071.
2) `make docs` doesn't work when you checkout v1.3.x branch on its
own. The scala docs step breaks the build. I tested a fix with one
update from the same Java/Scala docs PR I put in 4 days ago:
https://github.com/apache/incubator-mxnet/pull/13071/files#diff-c8ca1b4ea12114fbd742c76087bca0e4R121
This appears to fix the issue. (Scala version differences create
different artifacts which may cause that step to fail.) I can create a
PR for just this one change. I wouldn't apply #13071 - I tried it and
it failed with Java-related dependency errors. If we want to fix both,
then, maybe we can tackle that and the other errors and warnings that
were just introduced with the Java API too.
On Fri, Nov 16, 2018 at 8:44 PM sandeep krishnamurthy
 wrote:
>
> +1
>
> Build from source on Mac and Ubuntu 16.04.
>
> 1. Run GAN, DCGAN using Gluon on 1 GPU in Ubuntu and CPU Mac.
> 2. Convert a model to fp16, save and load in Gluon, perform inference.
>
> Best,
> Sandeep
>
> On Fri, Nov 16, 2018, 7:01 PM Indhu 
> > +1.
> >
> > Builds fine from source. Tested few CNN examples using Python binding.
> > Everything looks fine.
> >
> > Thanks,
> > Indu
> >
> >
> > On Fri, Nov 16, 2018, 4:16 PM Roshani Nagmote  > wrote:
> >
> > > +1
> > > Installed the release package on Ubuntu with CUDA, CUDNN enabled and ran
> > > train_cifar10.py example successfully.
> > >
> > > Thanks,
> > > Roshani
> > >
> > > On Fri, Nov 16, 2018 at 5:21 AM Anton Chernov 
> > wrote:
> > >
> > > > Thank you Carin, Steffen and Kellen for your votes.
> > > >
> > > > The results so far:
> > > >
> > > > * Binding *
> > > >
> > > > +1 votes:
> > > >   - Carin
> > > >
> > > > * Non-Binding *
> > > >
> > > > +1 votes:
> > > >   - Kellen
> > > >   - Steffen
> > > >
> > > > So far, we've got only positive votes, but unfortunately not enough to
> > > > conclude a result from the vote. I would like to remind everyone that
> > we
> > > > need at least 3 binding +1 votes. Therefore, the vote is extended until
> > > > Tuesday 20th of November 2018, 5pm CET (9am PT).
> > > >
> > > > I kindly ask the community to participate in voting for this patch
> > > release.
> > > >
> > > > Best regards
> > > > Anton
> > > >
> > > >
> > > > пт, 16 нояб. 2018 г. в 9:10, kellen sunderland <
> > > > kellen.sunderl...@gmail.com
> > > > >:
> > > >
> > > > > Just tested with 1.3.0 and those tests were failing for that release
> > as
> > > > > well.  Given it's not a regression I'm +1 (non-binding).
> > > > >
> > > > > On Thu, Nov 15, 2018 at 11:52 PM kellen sunderland <
> > > > > kellen.sunderl...@gmail.com> wrote:
> > > > >
> > > > > > Thanks for organizing the release Anton and for testing Carin and
> > > > > > Steffen.  Lots of great fixes in this release.  As we don't have
> > the
> > > > > > required 3 committers I'd suggest extending the vote for a few
> > days.
> > > > > >
> > > > > > I tested the following on MacOS 10.13, High Sierra:
> > > > > >
> > > > > > INCUBATING IN RELEASE FILE: check.
> > > > > > LICENSE check.
> > > > > > NOTICE check.
> > > > > > SIGNATURE check.
> > > > > > HASH check.
> > > > > > DISCLAIMER check.
> > > > > > SOURCE COMPILES VIA MAKEFILE check.
> > > > > > SOURCE COMPILES VIA CMAKE check.
> > > > > > C++ TESTS PASS fail
> > > > > > Two tests failing for me.
> > > > > > Build with flags: cmake -DUSE_CUDA=0 -DUSE_CUDNN=0 -DUSE_OPENMP=0
> > > > > > -DUSE_OPENCV=0 ..
> > > > > > Ran c++ tests with exclusions: ./tests/mxnet_unit_tests
> > > > > > --gtest_filter=-GpuTopology.*
> > > > > > Result:
> > > > > > [  FAILED  ] 2 tests, listed below:
> > > > > > [  FAILED  ] ACTIVATION_PERF.ExecuteBidirectional
> > > > > > [  FAILED  ] ACTIVATION_PERF.TimingCPU
> > > > > >
> > > > > > PYHTON UNIT TESTS PASS check.
> > > > > >
> > > > > > Not sure if the test failures are a regression so I'm +0
> > > (non-binding)
> > > > > >
> > > > > > On Thu, Nov 15, 2018 at 5:43 PM Steffen Rochel <
> > > > steffenroc...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > >> +1 build on MacOS Sierra following instructions on
> > > > > >>
> > > > > >>
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/MXNET/MXNet+Developer+Setup+on+Mac
> > > > > >> and run one training test.
> > > > > >>
> > > > > >> On Tue, Nov 13, 2018 at 2:34 PM Carin Meier  > >
> > > > > wrote:
> > > > > >>
> > > > > >> > +1 - Clojure package tested fine with Scala jars
> > > > > >> >
> > > > > >> > On Mon, Nov 12, 2018 at 6:53 PM Anton Chernov <
> > > mecher...@gmail.com>
> > > > > >> wrote:
> > > > > >> >
> > > > > >> > > Dear MXNet community,
> > > > > >> > >
> > > > > >> > > This is the vote to release Apache MXNet (incubating) version
> > > > 1.3.1.
> > > > > >> > Voting
> > > > > >> > > will start now, on Monday the 12th of 

Re: Nightly/Weekly tests for examples

2018-11-13 Thread Aaron Markham
Naveen - I agree with you there. Not every example should be moved to
tutorials... just the ones that would be great to have on the site and
to be tested regularly.
I think that having examples of specific APIs will be useful on the website too.

Ankit, there's a cost/benefit analysis for keeping both a command line
example and a notebook. That has maintenance overhead.
A couple of options to debate and apply on a case-by-case basis:
1) Only maintain one or the other.
2) If it's really important, maintain both, but update both codebases
with notes and directions for maintenance, so that if one is found
broken and is getting updated, contributors know that they should
update the other.

Since the CNN Viz example/tutorial is for visualization, I'd lean
towards #1 and keep only the notebook. Then delete the example, but
since it requires a python file from that examples folder as a
utility, move this to a tutorials_utils folder in /docs. That folder
should be organized appropriately and excluded in the docs build
config in conf.py. Look for "exclude_patterns".

My 3 cents... Cheers,
Aaron


On Tue, Nov 13, 2018 at 11:26 AM Khedia, Ankit
 wrote:
>
>
> We can start small consolidating few examples which are there in both 
> tutorials/examples folder.
> One of such example is GradCAM implementation:
> In tutorials - 
> https://github.com/apache/incubator-mxnet/blob/master/docs/tutorials/vision/cnn_visualization.md
> In examples- 
> https://github.com/apache/incubator-mxnet/blob/master/example/cnn_visualization
> The only difference is we have a python script in place of notebook in 
> examples folder for this particular example.
> Any thoughts/suggestions?
>
> On the other hand, I agree with Sandeep that there should be some basic 
> testing of examples and we can start with a small set of python examples to 
> begin with.
>
> —Ankit
>
>
> On Nov 13, 2018, at 10:31 AM, Naveen Swamy 
> mailto:mnnav...@gmail.com>> wrote:
>
> Aaron, IMO tutorials have a specific purpose, to introduce concepts and
> APIs to the users and I think converting examples to tutorials would
> overwhelm the users, we should carefully choose which examples we want to
> turn into tutorials.
>
> I agree that today examples are graveyard of untested code, my suggestion
> is to add some testing to the example when you touch the example - at the
> least to check the functionality. These can be run once a week.
>
>
> On Tue, Nov 13, 2018 at 6:52 AM Aaron Markham 
> mailto:aaron.s.mark...@gmail.com>>
> wrote:
>
> I've been actively promoting moving examples to tutorials during reviews.
> That way they fall under the testing umbrella and get added to the website.
>
> Many times there's not really a great distinction as to why something is in
> the examples folder, other than it's like a graveyard of untested sample
> code.
>
> I would suggest a starting strategy of when doing updates on examples, see
> if with just a little more effort, ask yourself, can it be converted to a
> tutorial?
>
> The last thing CI needs is more flaky tutorial tests, so whatever is done
> here should use the more robust approaches that are being discussed.
>
> Cheers,
> Aaron
>
> On Mon, Nov 12, 2018, 16:24 sandeep krishnamurthy <
> sandeep.krishn...@gmail.com<mailto:sandeep.krishn...@gmail.com> wrote:
>
> Thanks, Ankit for bringing this up. @Anirudh - All the concerns you
> raised
> are very valid. Here are my thoughts:
> 1. There were several examples that were crashing or had compiler errors.
> This is a very bad user experience. All example scripts should be at
> least runnable!
> 2. While I agree examples are too diverse (python scripts, notebooks,
> epochs, print statements etc..) We can always start small, we can start
> with 5 examples. We can use this to streamline all examples to be python
> scripts, print statements, with the main function invoker that can take
> params like epoch, dataset etc.
> 3. We can start with running weekly tests to avoid too long nightly test
> pipeline.
> 4. One possible issue can be on a few examples that depend on a large or
> controlled dataset. I am not sure yet, how to solve this, but, we can
> think.
>
> Any suggestions?
> Best,
> Sandeep
>
>
>
> On Mon, Nov 12, 2018 at 10:38 AM Anirudh Acharya 
> mailto:anirudhk...@gmail.com>>
> wrote:
>
> Hi Ankit,
>
> I have a few concerns about testing examples. Before writing tests for
> examples,
>
>   - you will need to first decide what constitutes a test for an
> example,
>   because examples are not API calls, which will have return
> statements
> and
>   the test can just call the API and assert for certain values. Just
> testing
>   if an example is a comp

Re: Nightly/Weekly tests for examples

2018-11-13 Thread Aaron Markham
I've been actively promoting moving examples to tutorials during reviews.
That way they fall under the testing umbrella and get added to the website.

Many times there's not really a great distinction as to why something is in
the examples folder, other than it's like a graveyard of untested sample
code.

I would suggest a starting strategy of when doing updates on examples, see
if with just a little more effort, ask yourself, can it be converted to a
tutorial?

The last thing CI needs is more flaky tutorial tests, so whatever is done
here should use the more robust approaches that are being discussed.

Cheers,
Aaron

On Mon, Nov 12, 2018, 16:24 sandeep krishnamurthy <
sandeep.krishn...@gmail.com wrote:

> Thanks, Ankit for bringing this up. @Anirudh - All the concerns you raised
> are very valid. Here are my thoughts:
> 1. There were several examples that were crashing or had compiler errors.
> This is a very bad user experience. All example scripts should be at
> least runnable!
> 2. While I agree examples are too diverse (python scripts, notebooks,
> epochs, print statements etc..) We can always start small, we can start
> with 5 examples. We can use this to streamline all examples to be python
> scripts, print statements, with the main function invoker that can take
> params like epoch, dataset etc.
> 3. We can start with running weekly tests to avoid too long nightly test
> pipeline.
> 4. One possible issue can be on a few examples that depend on a large or
> controlled dataset. I am not sure yet, how to solve this, but, we can
> think.
>
> Any suggestions?
> Best,
> Sandeep
>
>
>
> On Mon, Nov 12, 2018 at 10:38 AM Anirudh Acharya 
> wrote:
>
> > Hi Ankit,
> >
> > I have a few concerns about testing examples. Before writing tests for
> > examples,
> >
> >- you will need to first decide what constitutes a test for an
> example,
> >because examples are not API calls, which will have return statements
> > and
> >the test can just call the API and assert for certain values. Just
> > testing
> >if an example is a compilable python script will not add much value in
> > my
> >opinion.
> >- And testing for example output and results will require a re-write
> of
> >many of the examples, because many of them currently just have print
> >statements as outputs and does not return any value as such. I am not
> > sure
> >if it is worth the dev-effort.
> >- the current set of examples in the mxnet repo are very diverse -
> some
> >are written as python notebooks, some are just python scripts with
> paper
> >implementations, and some are just illustrations of certain mxnet
> > features.
> >I am curious to know how you will write tests for these things.
> >
> >
> > Looking forward to seeing the design of this test bed/framework.
> >
> >
> > Thanks
> > Anirudh Acharya
> >
> > On Fri, Nov 9, 2018 at 2:39 PM Marco de Abreu
> >  wrote:
> >
> > > Hello Ankit,
> > >
> > > that's a great idea! Using the tutorial tests as reference is a great
> > > starting point. If you are interested, please don't hesitate to attend
> > the
> > > Berlin user group in case you would like to discuss your first thoughts
> > > in-person before drafting a design.
> > >
> > > -Marco
> > >
> > >
> > > Am Fr., 9. Nov. 2018, 23:23 hat khedia.an...@gmail.com <
> > > khedia.an...@gmail.com> geschrieben:
> > >
> > > > Hi MXNet community,
> > > >
> > > > Recently, I and a few other contributors focussed on fixing examples
> in
> > > > our repository which were not working out of the box as expected.
> > > > https://github.com/apache/incubator-mxnet/issues/12800
> > > > https://github.com/apache/incubator-mxnet/issues/11895
> > > > https://github.com/apache/incubator-mxnet/pull/13196
> > > >
> > > > Some of the examples failed after API changes and remained uncaught
> > until
> > > > a user reported the issue. While the community is actively working on
> > > > fixing it, it might re-occur after few days if we don’t have a proper
> > > > mechanism to catch regressions.
> > > >
> > > > So, I would like to propose to enable nightly/weekly tests for the
> > > > examples similar to what we have for tutorials to catch any such
> > > > regressions. The test could check only basic functionalities/working
> of
> > > the
> > > > examples. It can run small examples completely whereas it can run
> long
> > > > training examples for only few epochs.
> > > >
> > > > Any thoughts from the community? Any other suggestions for fixing the
> > > same?
> > > >
> > > > Regards,
> > > > Ankit Khedia
> > > >
> > >
> >
>
>
> --
> Sandeep Krishnamurthy
>


Re: Run Sphinx checks on MXNet CI

2018-11-08 Thread Aaron Markham
Hi Anirudh,

Once the existing errors in docs building are cleaned up, I'm all for
having CI bubble up a build break when docs are broken by a PR. That
way we're keeping things up to date and not letting a minor bug turn
into a serious issue for the entire API documentation. One break
causes a ripple effect that can go unnoticed and then trying to find
it is like a needle in a haystack when there are already thousands of
warnings or errors that are being ignored.

We now have several docs troubleshooting tips in a documentation guide
thanks to the contributions of @frankfliu, @Roshrini, @vandanavk,
@vdantu, @vrakesh, and @zachgk.

This documentation guide is published on the developer cwiki:
https://cwiki.apache.org/confluence/display/MXNET/Documentation+Guide

I plan to continue to add to this guide as more PRs come in that
exhibit new ways of handling errors or warnings. That way, creating
docs and troubleshooting any build issues will be much easier.

Please let me know if you have any questions or feedback. You can
always add that directly to the wiki too.

Cheers,
Aaron


On Thu, Nov 8, 2018 at 11:21 AM Anirudh Acharya  wrote:
>
> Hi,
>
> Recently there was a barrage of issues related to documentation that was
> raised here -
> https://github.com/apache/incubator-mxnet/issues/created_by/aaronmarkham
> All the issues are related to Sphinx errors and warnings. These errors
> often lead to broken documentation. Ideally such errors should be caught
> before a PR gets merged, on the CI.
>
> Since we use Sphinx to generate the documentation for MXNet, can we have
> the CI run Sphinx tests on every PR so that we can preempt the problem of
> broken documentation.
>
> Any thoughts from the community? What might be involved to make this change
> to the CI?
>
>
> Regards
> Anirudh Acharya


Re: [Announce] Upcoming Apache MXNet (incubating) 1.3.1 patch release

2018-11-06 Thread Aaron Markham
Hi Anton,
I have the following suggestions for fixes to include in 1.3.1. These each
have updates to files that will impact docs generation for the 1.3.x
version of the website's Python API docs:

https://github.com/apache/incubator-mxnet/pull/12879
https://github.com/apache/incubator-mxnet/pull/12871
https://github.com/apache/incubator-mxnet/pull/12856

Thanks,
Aaron

On Tue, Nov 6, 2018 at 1:29 PM Lai Wei  wrote:

> Hi Anton,
>
> Thanks for driving this, I would like to include the following fix in
> 1.3.1:
> Allow infer shape partial on foreach operator:
> https://github.com/apache/incubator-mxnet/pull/12471
>
> Keras-MXNet needs this functionality to infer shape partially
> on foreach operator. (Used in RNN operators)
>
> Thanks a lot!
>
>
> Best Regards
> Lai Wei
>
>
>
> On Tue, Nov 6, 2018 at 10:44 AM Haibin Lin 
> wrote:
>
> > Hi Naveen and Anton,
> >
> > Thanks for pointing that out. You are right that these are not critical
> > fixes. Putting them in 1.4.0 is more appropriate. PRs are closed.
> >
> > Best,
> > Haibin
> >
> > On Tue, Nov 6, 2018 at 7:35 AM Naveen Swamy  wrote:
> >
> > > Please note that this is a patch release(1.3.1) to address critical
> > bugs!,
> > > For everything else please wait for 1.4.0 which is planned very shortly
> > > after 1.3.1
> > >
> > > > On Nov 6, 2018, at 7:17 AM, Anton Chernov 
> wrote:
> > > >
> > > > The following PR's have been created so far:
> > > >
> > > > Infer dtype in SymbolBlock import from input symbol (v1.3.x)
> > > > https://github.com/apache/incubator-mxnet/pull/13117
> > > >
> > > > [MXNET-953] Fix oob memory read (v1.3.x)
> > > > https://github.com/apache/incubator-mxnet/pull/13118
> > > >
> > > > [MXNET-969] Fix buffer overflow in RNNOp (v1.3.x)
> > > > https://github.com/apache/incubator-mxnet/pull/13119
> > > >
> > > > [MXNET-922] Fix memleak in profiler (v1.3.x)
> > > > https://github.com/apache/incubator-mxnet/pull/13120
> > > >
> > > > Set correct update on kvstore flag in dist_device_sync mode (v1.3.x)
> > > > https://github.com/apache/incubator-mxnet/pull/13121
> > > >
> > > > update mshadow (v1.3.x)
> > > > https://github.com/apache/incubator-mxnet/pull/13122
> > > >
> > > > CudnnFind() usage improvements (v1.3.x)
> > > > https://github.com/apache/incubator-mxnet/pull/13123
> > > >
> > > > Fix lazy record io when used with dataloader and multi_worker > 0
> > > (v1.3.x)
> > > > https://github.com/apache/incubator-mxnet/pull/13124
> > > >
> > > >
> > > > As stated previously I would be rather opposed to have following PR's
> > it
> > > in
> > > > the patch release:
> > > >
> > > > Gluon LSTM Projection and Clipping Support (#13055) v1.3.x
> > > > https://github.com/apache/incubator-mxnet/pull/13129
> > > >
> > > > sample_like operators (#13034) v1.3.x
> > > > https://github.com/apache/incubator-mxnet/pull/13130
> > > >
> > > >
> > > > Best
> > > > Anton
> > > >
> > > > вт, 6 нояб. 2018 г. в 16:06, Anton Chernov :
> > > >
> > > >> Hi Haibin,
> > > >>
> > > >> I have a few comments regarding the proposed performance improvement
> > > >> changes.
> > > >>
> > > >> CUDNN support for LSTM with projection & clipping
> > > >> https://github.com/apache/incubator-mxnet/pull/13056
> > > >>
> > > >> There is no doubt that this change brings value, but I don't see it
> > as a
> > > >> critical bug fix. I would rather leave it for the next major
> release.
> > > >>
> > > >> sample_like operators
> > > >> https://github.com/apache/incubator-mxnet/pull/13034
> > > >>
> > > >> Even if it's related to performance, this is an addition of
> > > functionality
> > > >> and I would also push this to be in the next major release only.
> > > >>
> > > >>
> > > >> Best
> > > >> Anton
> > > >>
> > > >>
> > > >> вт, 6 нояб. 2018 г. в 15:55, Anton Chernov :
> > > >>
> > > >>> Hi Patric,
> > > >>>
> > > >>> This change was listed in the 'PR candidates suggested for
> > > consideration
> > > >>> for v1.3.1 patch release' section [1].
> > > >>>
> > > >>> You are right, I also think that this is not a critical hotfix
> change
> > > >>> that should be included into the 1.3.1 patch release.
> > > >>>
> > > >>> Thus I'm not making any further efforts to bring it in.
> > > >>>
> > > >>> Best
> > > >>> Anton
> > > >>>
> > > >>> [1]
> > > >>>
> > >
> >
> https://cwiki.apache.org/confluence/display/MXNET/Project+Proposals+for+next+MXNet+Release#PR_candidates
> > > >>>
> > > >>>
> > > >>> вт, 6 нояб. 2018 г. в 1:14, Zhao, Patric :
> > > >>>
> > >  Hi Anton,
> > > 
> > >  Thanks for looking into the MKL-DNN PR.
> > > 
> > >  As my understanding of cwiki (
> > > 
> > >
> >
> https://cwiki.apache.org/confluence/display/MXNET/Project+Proposals+for+next+MXNet+Release
> > >  ),
> > >  these features will go into 1.4 rather than patch release of
> 1.3.1.
> > > 
> > >  Feel free to correct me :)
> > > 
> > >  Thanks,
> > > 
> > >  --Patric
> > > 
> > > > -Original Message-
> > > > From: Anton Chernov 

Re: [VOTE] - Adopt "Become a Committer and PPMC Member" Document

2018-10-30 Thread Aaron Markham
+1 non-binding
One minor thing first: can you define PPMC in the doc? It's brought in
without saying what it stands for. Even the link it goes to just talks
about PMC and there's no mention of PMCC... so I'm not sure what the
definition is.


On Tue, Oct 30, 2018 at 8:07 AM Carin Meier  wrote:

> From the feedback in the thread. I changed the wording of "Privileges" to
> "Rights and Responsibilities".
>
> If I misunderstood anything, please let me know.
>
> Best,
> Carin
>
> On Tue, Oct 30, 2018 at 7:00 AM Sergio Fernández 
> wrote:
>
> > +1 since the Beam model is much more open than the current one.
> >
> > Here my two cents to the discussion:
> >
> > You can see that in the past was different,, but we had evolved as
> > foundation. As general recommendation, the new way is to spend less
> effort
> > in ad-hoc bylaws on every project/podling and adopt the general ones. The
> > easier the project is managed, normally the better the community evolves.
> >
> > In addition, as a linguistic detail: commiters and/or pmc do not have
> more
> > "privileges", but "responsibilities".
> >
> > Cheers,
> >
> > On Mon, Oct 29, 2018, 15:47 Carin Meier  wrote:
> >
> > > This vote is to adopt the document
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/MXNET/Become+an+Apache+MXNet+%28incubating%29+Committer+and+PPMC+Member+Proposal
> > > to replace the current document
> > > https://cwiki.apache.org/confluence/display/MXNET/Becoming+a+Committer
> > >
> > > The dev discussion thread is here
> > >
> > >
> >
> https://lists.apache.org/thread.html/e61ffa26af374de7a99c475d406e462a00b26cfc1155e232198dd53e@%3Cdev.mxnet.apache.org%3E
> > >
> > > The vote will be a procedural issue vote as defined
> > > https://www.apache.org/foundation/voting.html
> > >
> > > Votes on procedural issues follow the common format of majority rule
> > unless
> > > otherwise stated. That is, if there are more favourable votes than
> > > unfavourable ones, the issue is considered to have passed -- regardless
> > of
> > > the number of votes in each category. (If the number of votes seems too
> > > small to be representative of a community consensus, the issue is
> > typically
> > > not pursued. However, see the description of lazy consensus
> > >  for a
> > > modifying factor.)
> > >
> > > The vote will run until Friday Nov 2nd at 6:00 am EST
> > >
> > > Thanks,
> > > Carin
> > >
> >
>


Python API docs issues

2018-10-19 Thread Aaron Markham
Hello everyone!

I'm writing this note to raise awareness about some issues I'm seeing with
the Python API docs. While some of the issues can be traced back prior to
v1.3.0, the biggest issue of docs not rendering is fairly recent.
At a high level, Sphinx (our Python API docs generator) is getting confused
with a couple of things:
1. Folder changes (moving optimizer.py to /optimizer/, and io.py to /io/,
etc.)
2. Ambiguous references (where ndarray, type, shape, name, etc are declared
in different modules)

This is leading the Python docs to link up to the wrong place, or not link
up at all, or not generate at all.
If you didn't know, most of the API reference is auto-generated with about
two lines of rst that invokes Sphinx to do its thing. Except now Sphinx is
not doing what it's supposed do.

I've done a couple of workaround to fix the docs in areas that were
reported broken - by manually specifying the members of the API that should
be generated and manually linking up parameters, but I get this nagging
feeling that this could be fixed upstream. The init functions, the
importing, or ...? I don't really know, but Sphinx is unhappy and the docs
are suffering. Or maybe since the project is growing, we have give Sphinx a
little more TLC and be very good at being specific in the documentation.

I've documented what I've been seeing in some recent PRs and issues.

* Start here with the Python IO API since I've provided several links to
tour the two "Sphinx confusion" issues I previously mentioned.
https://github.com/apache/incubator-mxnet/pull/12871

* I also reported Sphinx not being able to access shorthand methods in
other PR where I fixed the test_utils and visualization docs. Looks at the
comments.
https://github.com/apache/incubator-mxnet/pull/12455
Note that mxnet.viz doesn't work but mxnet.visualization does... for
Sphinx. But everyone else can use mxnet.viz. What's up with that? I found
this pretty strange.

* This happened again with the ONNX import_model function. I deployed a
workaround so the ONNX docs were at least functional. It is reported here:
https://github.com/apache/incubator-mxnet/issues/12318

* Ndarray docs are messed up right now. Reported here:
https://github.com/apache/incubator-mxnet/issues/12829

I can probably fix the Ndarray API docs using the same pattern I used for
the IO API, but I don't want to go down this path too far when there might
be a simple solution. Otherwise, it seems pretty clear that every .py file
and every .md file for the Python API will need some updating.

Thoughts? Any Sphinx whisperers out there?

Thanks!
Aaron


Re: New MXNet CI guide: Managing and accessing credentials

2018-10-17 Thread Aaron Markham
Thanks Marco, this is a great feature. I didn't see git listed under "when
to use this guide". Can we use this to access git credentials like for when
we publish the website?
Cheers,
Aaron

On Wed, Oct 17, 2018, 06:19 Marco de Abreu
 wrote:

> Hello,
>
> I have just published a new guide how to manage and access credentials in
> MXNet CI. It's available at
>
> https://cwiki.apache.org/confluence/display/MXNET/Managing+and+accessing+credentials
> .
>
> This is was part of the preparation for the Scala Maven Publish job that
> will run as Continuous Deployment.
>
> Best regards,
> Marco
>


Re: Which merge option to use on the Import Julia binding PR?

2018-09-26 Thread Aaron Markham
+1 to rebase and merge to retain the efforts of all of the contributors. If
there's some git maintenance that can trim it down from 700+ commits then
maybe that's a compromise.

On Wed, Sep 26, 2018, 21:23 Naveen Swamy  wrote:

> this PR comes from more than 1 individual, if we squash merge we'll not be
> able to attribute the contribution of those individuals.
>
> +1 to rebase merge to preserve history
>
> On Thu, Sep 27, 2018 at 12:04 AM, Tianqi Chen 
> wrote:
>
> > One of the main reason for a rebase merge is that it preserves the commit
> > history of the MXNet.jl package contributors, and given that the project
> > has been evolved since 2015 and has always been a high-quality language
> > module for MXNet.
> >
> > I think we should take an exception here to preserve the commit history
> of
> > each individual contributors to the Julia binding and welcome them to the
> > community.
> >
> > Tianqi
> >
> > On Wed, Sep 26, 2018 at 8:55 PM Tianqi Chen 
> > wrote:
> >
> > > In this particular case, I would suggest rebase and merge.
> > >
> > > The main reasoning is that the commit log of the Julia binding is not
> > > simple WIP commits, every commit there has been done through testcases
> > and
> > > it is important for us to respect the developer of the effort. It is
> also
> > > good to trace back the history of the commits more easily.
> > >
> > > Tianqi
> > >
> > >
> > > Tianqi
> > >
> > > On Wed, Sep 26, 2018 at 5:34 PM Carin Meier 
> > wrote:
> > >
> > >> Chiyuan,
> > >>
> > >> Thanks for the prompt to find some clarity of the pros and cons of
> > each. I
> > >> think that will help drive us to the right decision. I think some of
> > those
> > >> reasons are the ones you listed. I will take a stab below at outlining
> > >> what
> > >> I see. Feel free to chime in if I missed any.
> > >>
> > >> *Squash and Merge*
> > >>   *Pros* - It is the project standard
> > >>   - It will provide one commit for the feature and lessen the
> > need
> > >> for 700+ commits rebased on top of master.
> > >>  - It is easier for a user to do git log to browse commits and
> > see
> > >> what was features were added.
> > >>   *Cons* - I don't know how github would handle squashing all those
> > commit
> > >> messages into one. Will it be too much?
> > >> - You lose the granularity of the features individual
> > commits
> > >>
> > >> *Rebase and Merge*
> > >>  * Pros *- You don't have a huge commit message with one commit
> > >>   -  You do have the granularity of the individual features of
> > the
> > >> commit
> > >>  * Cons *- It is not the project standard
> > >>- You have 700+ commits on top of master that might be
> harder
> > >> to
> > >> see the ones that went in right before. (like someone browsing
> commits)
> > >>
> > >> On Wed, Sep 26, 2018 at 8:12 PM Chiyuan Zhang 
> > wrote:
> > >>
> > >> > Hi Carin,
> > >> >
> > >> > Can you clarify the pros and cons of the two approaches? Is the main
> > >> > concern here about logistics (e.g. preserving the history of the
> > >> original
> > >> > repo and developments) or technical issue (e.g. using squash might
> end
> > >> up
> > >> > with a hge commit message that might be difficult or hard to
> > >> handle)?
> > >> >
> > >> > I think it might not be very likely that someone is going to cherry
> > pick
> > >> > revert some of the commits. But preserving the commit history is
> still
> > >> > useful in case one need to trace the change or bisect for some
> > >> regression
> > >> > bugs, etc.
> > >> >
> > >> > Just to provide some context: the PR actually contains 700+ commits,
> > >> and it
> > >> > dates back to 2015. The development of the Julia binding started in
> > the
> > >> > early stage of MXNet. We started with a separate repo due to the
> > >> > requirement of the package system of julia.
> > >> >
> > >> > Best,
> > >> > Chiyuan
> > >> >
> > >> > On Wed, Sep 26, 2018 at 3:41 PM Carin Meier 
> > >> wrote:
> > >> >
> > >> > > The Import Julia binding PR ,(
> > >> > > https://github.com/apache/incubator-mxnet/pull/10149), is getting
> > >> very
> > >> > > close to being merged. Because of the large number of commits
> there
> > >> was a
> > >> > > suggestion not to use the usual "Squash and Merge".  The only
> option
> > >> > would
> > >> > > be "Rebase and Merge" since merging with a merge commit is not
> > enabled
> > >> > for
> > >> > > the project.
> > >> > >
> > >> > > *Squash and Merge* - The commits from this branch will be combined
> > >> into
> > >> > one
> > >> > > commit in the base branch (With all the commit messages combined)
> > >> > >
> > >> > > *Rebase and Merge* - The commits from this branch will be rebased
> > and
> > >> > added
> > >> > > to the base branch
> > >> > >
> > >> > > The PR is over 250+ commits (Github won't show all of them)
> > >> > >
> > >> > > Thoughts about how we should handle the merge?
> > >> > >
> > >> > > Thanks,
> > >> > > Carin
> > >> > >
> > >> >
> > >>
> > >
> >
>


Re: [LAZY VOTE] Consolidating developer guide in one place (cwiki preferred)

2018-09-26 Thread Aaron Markham
I think the latest feedback has been great. It seems to be mostly user
level issues though. Installation and usage primarily, with a sprinkle of
*if that stuff was better then I might be able to contribute*.

I've (with a few other contributors) tackled some of the very direct bits
of feedback for the website by incremental improvement of the install
pages, Gluon info, and UX for the API docs.

I've started additional planning for updates by adding an epic with
specific stories and tasks to Jira for the documentation pipeline (the
backend part of the website build):
https://issues.apache.org/jira/browse/MXNET-957

I've also added one that is more specific to the website's content:
https://issues.apache.org/jira/browse/MXNET-986
This is where I've captured only two tasks related to transitioning content
related to "contributing to MXNet" over to the wiki. Any pointers on which
content to move would help. These could be added as tasks too.

I welcome any suggestions, additions, and contributions to either of these
epics.

Cheers,
Aaron

On Wed, Sep 26, 2018, 00:02 Lin Yuan  wrote:

> Hi Aaron,
>
> Do we have a resolution for this proposal yet? Recently, there have been
> many asks for a better documentation for MXNet developers. I think it's a
> good time that we consolidate the developer documentation in a central
> place. Any thoughts or plan?
>
> Many Thanks,
>
> Lin
>
> On Tue, Sep 4, 2018 at 1:55 PM Lin Yuan  wrote:
>
> > +1
> >
> > On Tue, Sep 4, 2018 at 1:46 PM Aaron Markham 
> > wrote:
> >
> >> I'd like to call for a lazy vote on this before proceeding. Already had
> >> some +1s but let's be sure.
> >>
> >> The vote is to move developer guide info to cwiki. User guides would
> >> remain
> >> on the website.
> >>
> >> On Tue, Aug 21, 2018 at 12:53 PM sandeep krishnamurthy <
> >> sandeep.krishn...@gmail.com> wrote:
> >>
> >> > +1
> >> > Thanks Lin and Aaron. I agree website to cover all user facing
> >> > documentation and a separate consolidated and organized developer
> >> focussed
> >> > docs in one place (cwiki).
> >> >
> >> >
> >> > Note: Permissions on cwiki is currently not well managed with many
> >> people
> >> > having full admin rights to edit/create/delete pages. Should be fine
> for
> >> > now, but, when we start accumulating many documents and resources, we
> >> > should probably revisit on Delete permissions.
> >> >
> >> >
> >> > On Tue, Aug 21, 2018 at 11:57 AM Lin Yuan 
> wrote:
> >> >
> >> > > Hi Aaron,
> >> > >
> >> > > Thanks for your answer. I think it's a very worthwhile effort to
> move
> >> all
> >> > > the developer related content from mxet.io website to a dedicated
> >> > > developer
> >> > > site. Would you like to initiate this effort?
> >> > >
> >> > > Best,
> >> > >
> >> > > Lin
> >> > >
> >> > > On Wed, Aug 15, 2018 at 3:47 PM Haibin Lin <
> haibin.lin@gmail.com>
> >> > > wrote:
> >> > >
> >> > > > +1
> >> > > >
> >> > > > On Wed, Aug 15, 2018 at 1:10 PM, Aaron Markham <
> >> > > aaron.s.mark...@gmail.com>
> >> > > > wrote:
> >> > > >
> >> > > > > Hi Lin, I agree with this organization. If you feel like
> >> somethings
> >> > > > should
> >> > > > > be transitioned from the website to the wiki, I can help with
> >> that,
> >> > but
> >> > > > for
> >> > > > > the moment I've been suggesting that new developer-focused
> >> content be
> >> > > > > placed on the wiki.
> >> > > > >
> >> > > > > On Tue, Aug 14, 2018 at 10:40 AM, Lin Yuan  >
> >> > > wrote:
> >> > > > >
> >> > > > > > Dear MXNet community,
> >> > > > > >
> >> > > > > > As a developer, I noticed we have some developer guide
> >> scattered in
> >> > > > > > different websites (mxnet.io, cwiki):
> >> > > > > >
> >> > > > > > E.g.
> >> > > > > >
> >> > > > > > How to Create New Operators (Layers): [
> >> > > > > > https://mxnet.incubator.apache.org/faq/new_op.html]
> >> > > > > > A Guide to Implementing Sparse Operators in MXNet Backend [
> >> > > > > > https://cwiki.apache.org/confluence/display/MXNET/A+
> >> > > > > > Guide+to+Implementing+Sparse+Operators+in+MXNet+Backend
> >> > > > > > ]
> >> > > > > >
> >> > > > > > When searching developer guide by keyword, only one of them
> can
> >> be
> >> > > > > returned
> >> > > > > > on either site.
> >> > > > > >
> >> > > > > > It will be more convenient for developers if all the developer
> >> > guide
> >> > > > > > resides on cwiki and all user guide (non-developer) on the
> >> > mxnet.io
> >> > > > > > website. We can add a link on mxnet.io to refer all
> developers
> >> to
> >> > > > cwiki
> >> > > > > > for
> >> > > > > > guidance.
> >> > > > > >
> >> > > > > > Any comment is appreciated.
> >> > > > > >
> >> > > > > > Best Regards,
> >> > > > > >
> >> > > > > > Lin
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >> >
> >> > --
> >> > Sandeep Krishnamurthy
> >> >
> >>
> >
>


Re: improving API docs and tutorials user experience

2018-09-21 Thread Aaron Markham
I fixed the most annoying behavior with the versions dropdown here:
https://github.com/apache/incubator-mxnet/pull/12632
Now it'll remember what doc you're on when you switch versions.
I show an example with autograd where when you switch versions it keeps you
on topic even down to the bookmark. If you go to an old version where it
doesn't exist you get a custom 404.
I guess it could be possible to daisy chain some checks and redirects, but
this could be jarring.
I like this concept of suggestions though. What if the 404 page gets a
query string with a topic payload, and we use that to throw out some search
results?




On Thu, Sep 20, 2018 at 1:57 PM Afrooze, Sina  wrote:

> This is not a fully baked idea, so bear with me. Instead of using regex,
> what if we use search techniques to locate the page user wants to look at.
> If I'm currently looking at mxnet.ndarray.convolution operator in master,
> take me to the same operator API in 1.2.1, regardless of whether it is in a
> similar page or whether we decided to re-organize API pages and create a
> page just for convolution. Likewise for tutorials (i.e. find the tutorial
> that best matches the content I'm looking at, even if it's been renamed or
> moved around). I feel like it can be a great internship project. - Sina
>
> On 9/18/18, 5:08 PM, "Aaron Markham"  wrote:
>
> Hello dev list!
>
> I've been doing a bit with .htaccess redirects on the site to get us
> things
> like custom 404s and redirecting google searches that come in on
> outdated
> material. That's working pretty well because the redirects are simple.
>
> I need help though. I've written a short proposal on how I think the
> UX for
> API docs and tutorials should work. I'm not trying a major jump here.
> This
> is a small amount of change for a better experience.
> For example, right now, if you're browsing API docs for 1.3.0 and want
> to
> see the master version, you would hit the dropdown and select master.
> Then
> you get taken to the home page for master and have to go find the
> document
> again. (It's always done this.) It should stay on that document, but
> give
> the latest version. Or for tutorials, if you request a really old,
> likely
> broken tutorial because it happens to show up in google search
> results, the
> site should kindly escort you to the latest, tested tutorial.
>
> mod_rewrite and .htaccess can do this, but it requires regex skills
> that I
> lack. I've tried hundreds of variations now, and would like some
> guidance.
>
> Here's the proposal and my notes:
>
> https://cwiki.apache.org/confluence/display/MXNET/Version+calls+and+redirects+for+Tutorials+and+API+documents
>
> I appreciate your help with this!
>
> Cheers,
> Aaron
>
>
>
>


Re: Remove MKLML as dependency

2018-09-20 Thread Aaron Markham
I find it unintuitive that mxnet-mkl doesn't actually ship with MKL. Why
isn't it called mxnet-mkldnn instead?

Side note, if mkldnn fulfills BLAS requirements, then why can't we strip
out OpenBLAS for the "mxnet-mkl" package? Is there no way to make the
submodules conform to using mkldnn? All in the spirit of simplifying
things...limiting the deps...

On Sep 20, 2018 07:41, "Lv, Tao A"  wrote:

Hah, seems it's a little confusing here. I think the "Intel MKL" in the
first statement includes both the full MKL and MKLML library. And the
"dynamic library" there obviously means the MKLML which is delivered in
MKL-DNN repo.

MKLML is a subset of full MKL and includes all BLAS functions for both
single precision and double precision. From this point of view, I think it
can be used as a BLAS library, but cannot be used as full MKL.


-tao

-Original Message-
From: Chris Olivier [mailto:cjolivie...@gmail.com]
Sent: Thursday, September 20, 2018 9:36 PM
To: dev@mxnet.incubator.apache.org
Subject: Re: Remove MKLML as dependency

thanks for the info. I am still a little confused — your statement said
“MKL” and not “MKLML”, so my question is still the same.  Are GEMMS in
MKLML or just MKL? I know MKLML doesn’t have a blas library like the main
MKL.

On Wed, Sep 19, 2018 at 11:49 PM Lv, Tao A  wrote:

> Hi Chris, please kindly check the statements here:
> https://github.com/intel/mkl-dnn#installation
>
> " Intel MKL-DNN can take advantage of optimized matrix-matrix
> multiplication (GEMM) function from Intel MKL. The dynamic library
> with this functionality is included in the repository. "
>
> " You can choose to build Intel MKL-DNN without binary dependency. The
> resulting version will be fully functional, however performance of
> certain convolution shapes and sizes and inner product relying on
> SGEMM function may be suboptimal."
>
> -tao
>
> -Original Message-
> From: Chris Olivier [mailto:cjolivie...@gmail.com]
> Sent: Thursday, September 20, 2018 11:20 AM
> To: dev@mxnet.incubator.apache.org
> Subject: Re: Remove MKLML as dependency
>
> maybe I missed it, but what does MKLML have that mkldnn doesn’t have
> that makes it necessary?
>
> what’s the motivation for removing it?
>
> On Tue, Sep 18, 2018 at 11:31 PM Lv, Tao A  wrote:
>
> > If you just want to test the performance, I think you need link MKL
> > for BLAS and MKL-DNN for NN. Also MKL-DNN should link MKL for better
> > performance.
> >
> > Here are some ways for you to install full MKL library if you don't
> > have
> > one:
> > 1. Register and download from intel website:
> > https://software.intel.com/en-us/mkl
> > 2. Apt-get/yum: currently it need configure Intel’s repositories.
> > a.
> >
> https://software.intel.com/en-us/articles/installing-intel-free-libs-a
> nd-python-yum-repo
> > b. https://software.intel.com/en-us/articles/
> > thatinstalling-intel-free-libs-and-python-apt-repo
> >  > s-
> > and-python-apt-repo> 3. pip install mkl / mkl-devel: ‘mkl’ package
> > and-python-apt-repo> has

> > the runtime and ‘mkl-devel’ includes everything with the headers
> > a.
> > https://software.intel.com/en-us/articles/installing-the-intel-distr
> > ib ution-for-python-and-intel-performance-libraries-with-pip-and

> > 4. conda install: also has mkl and mkl-devel
> > a. https://anaconda.org/intel/mkl
> > b. https://anaconda.org/intel/mkl-devel
> >
> > If you want to redistribute MKL with MXNet, you may need take care
> > of the license issue. Currently, MKL is using ISSL (
> > https://software.intel.com/en-us/license/intel-simplified-software-l
> > ic
> > ense
> > ).
> >
> > -Original Message-
> > From: Zai, Alexander [mailto:alex...@amazon.com.INVALID]
> > Sent: Wednesday, September 19, 2018 12:49 PM
> > To: dev@mxnet.incubator.apache.org
> > Subject: Re: Remove MKLML as dependency
> >
> > Will test it out tomorrow.
> >
> > On the side, what is the best way to test MKL build for MXnet. MKL
> > is licensed?
> >
> > Best,
> > Alex
> >
> > On 9/18/18, 7:50 PM, "Lv, Tao A"  wrote:
> >
> > Hi Alex,
> >
> > Thanks for bringing this up.
> >
> > The original intention of MKLML is to provide a light and
> > easy-to-access library for ML/DL community. It's released with
> > MKL-DNN under Apache-2.0 license.
> >
> > AFAIK, MKL-DNN still relies on it for better performance. So I'm
> > afraid there will be a performance regression in MKL pip packages if
> > MKLML is simply removed.
> >
> > Have you ever tried the build without MKLML and how does the
> > performance look like?
> >
> > -tao
> >
> > -Original Message-
> > From: Alex Zai [mailto:aza...@gmail.com]
> > Sent: Wednesday, September 19, 2018 4:49 AM
> > To: dev@mxnet.incubator.apache.org
> > Subject: Remove MKLML as dependency
> >
> > On our build from source page we have a list of blas libraries
> > that are recommended:
> >
> 

Re: multiple installation guides?

2018-09-20 Thread Aaron Markham
Sure...
~~
Apache Infra,
Please remove the /test/ directory found at
https://mxnet.incubator.apache.org/test
This directory is not found in the
https://github.com/apache/incubator-mxnet-site repository, yet it has web
traffic going to this neglected folder.
Are there any other folders on the web server that might also fit in this
category? If so, please advise and remove if they appear to be similar
"test" folders.

Thanks,
Aaron

On Thu, Sep 20, 2018 at 6:00 AM Marco de Abreu
 wrote:

> Sure. Aaron, can you write something down that can be copy and pasted into
> a ticket so that our mentors just have to create it?
>
> I think we could ask Infra for an investigation about *why* this happened,
> considering the fact that it should be an exact mirror, and tools to
> self-service these kind of things.
>
> -Marco
>
> On Wed, Sep 19, 2018 at 4:11 PM Aaron Markham 
> wrote:
>
> > It's not on the site repo. Seems like it is only on the Apache infra. Can
> > someone request it's removal?
> >
> > On Tue, Sep 18, 2018, 20:34 Hagay Lupesko  wrote:
> >
> > > The /test site seems to be something old that should have been removed
> a
> > > long time ago, it lists versions 0.10 and 0.10.14 :)
> > > Maybe Aaron has an idea what needs to be done to remove it...
> > >
> > > On Fri, Sep 14, 2018 at 4:55 PM Alex Zai  wrote:
> > >
> > > > Why do we have two sets of installation guides?
> > > >
> > > > http://mxnet.incubator.apache.org/test/get_started/install.html
> > > >
> > > >
> > >
> >
> https://mxnet.incubator.apache.org/install/index.html?platform=Linux=Python=CPU
> > > >
> > > > The /test domain is also not secure. If this is not suppose to be
> > > > public we should remove this as it is confusing.
> > > >
> > >
> >
>


Re: Some feedback from MXNet Zhihu topic

2018-09-19 Thread Aaron Markham
Thanks for this translation and feedback Qing!
I've addressed point 3 of the documentation feedback with this PR:
https://github.com/apache/incubator-mxnet/pull/12604
I'm not sure how to take the first two points without some explicit URLs
and examples, so if anyone has those I'd be happy to take a look if there's
some glitch vs missing or wrong docs.

Also, I would agree that there should be some more simple examples. Often
times the examples are too complicated and unclear about what is important
or not. The audience targeting is for deep learning practitioners, not
"newbies".

And on a related note, I'd really like to pull the Gluon stuff into the API
section. It's confusing as its own navigation item and orphaned
information. It could have a navigation entry at the top of the API list
like "Python: Gluon" or just "Gluon" then list "Python: Module" or just
"Python". Or running this the other way, the Gluon menu could have API and
Tutorials and be more fleshed out, though this is not my preference. Either
way, it needs some attention.

Cheers,
Aaron

On Wed, Sep 19, 2018 at 11:04 AM Qing Lan  wrote:

> Hi all,
>
> There was a trend topic in
> Zhihu (a famous Chinese Stackoverflow+Quora) asking about the status of
> MXNet in 2018 recently. Mu replied the thread and obtained more than 300+
> `like`.
> However there are a few concerns addressed in the comments of this thread,
> I have done some simple translation from Chinese to English:
>
> 1. Documentations! Until now, the online doc still contains:
> 1. Depreciated but not updated doc
> 2. Wrong documentation with poor description
> 3. Document in Alpha stage such as you must install `pip
> –pre` in order to run.
>
> 2. Examples! For Gluon specifically, many examples are still mixing
> Gluon/MXNet apis. The mixure of mx.sym, mx.nd mx.gluon confused the users
> of what is the right one to choose in order to get their model to work. As
> an example, Although Gluon made data encapsulation possible, still there
> are examples using mxn.io.ImageRecordIter with tens of params (feels like
> gluon examples are simply the copy from old Python examples).
>
> 3. Examples again! Comparing to PyTorch, there are a few examples I don't
> like in Gluon:
> 1. Available to run however the code structure is still
> very complicated. Such as example/image-classification/cifar10.py. It
> seemed like a consecutive code concatenation. In fact, these are just a
> series of layers mixed with model.fit. It makes user very hard to
> modify/extend the model.
> 2. Only available to run with certain settings. If users
> try to change a little bit in the model, crashes will happen. For example,
> the multi-gpu example in Gluon website, MXNet hide the logic that using
> batch size to change learning rate in a optimizer. A lot of newbies didn't
> know this fact and they would only find that the model stopped converging
> when batch size changed.
> 3. The worst scenario is the model itself just simply
> didn't work. Maintainers in the MXNet community didn't run the model (even
> no integration test) and merge the code directly. It makes the script not
> able run till somebody raise the issues and fix it.
>
> 4. The Community problem. The core advantage for MXNet is it's scalability
> and efficiency. However, the documentation of some tools are confusing.
> Here are two examples:
>
> 1. im2rec contains 2 versions, C++ (binary) and python.
> But nobody would thought that the argparse in these tools are different (in
> the meantime, there is no appropriate examples to compare with, users could
> only use them by guessing the usage).
>
> 2. How to combine MXNet distributed platform with
> supercomputing tool such as Slurm? How do we do profiling and how to debug.
> A couples of companies I knew thought of using MXNet for distributed
> training. Due to lack of examples and poor support from the community, they
> have to change their models into TensorFlow and Horovod.
>
> 5. The heavy code base. Most of the MXNet examples/source
> code/documentation/language binding are in a single repo. A git clone
> operation will cost tens of Mb. The New feature PR would takes longer time
> than expected. The poor reviewing response / rules keeps new contributors
> away from the community. I remember there was a call for
> document-improvement last year. The total timeline cost a user 3 months of
> time to merge into master. It almost equals to a release interval of
> Pytorch.
>
> 6. To Developers. There are very few people in the community discussed the
> improvement we can take to make MXNet more user-friendly. It's been so easy
> to trigger tens of stack issues during coding. Again, is that a requirement
> for MXNet users to be familiar with C++? The connection between Python and
> C lacks a IDE lint (maybe MXNet assume every 

Re: multiple installation guides?

2018-09-19 Thread Aaron Markham
It's not on the site repo. Seems like it is only on the Apache infra. Can
someone request it's removal?

On Tue, Sep 18, 2018, 20:34 Hagay Lupesko  wrote:

> The /test site seems to be something old that should have been removed a
> long time ago, it lists versions 0.10 and 0.10.14 :)
> Maybe Aaron has an idea what needs to be done to remove it...
>
> On Fri, Sep 14, 2018 at 4:55 PM Alex Zai  wrote:
>
> > Why do we have two sets of installation guides?
> >
> > http://mxnet.incubator.apache.org/test/get_started/install.html
> >
> >
> https://mxnet.incubator.apache.org/install/index.html?platform=Linux=Python=CPU
> >
> > The /test domain is also not secure. If this is not suppose to be
> > public we should remove this as it is confusing.
> >
>


improving API docs and tutorials user experience

2018-09-18 Thread Aaron Markham
Hello dev list!

I've been doing a bit with .htaccess redirects on the site to get us things
like custom 404s and redirecting google searches that come in on outdated
material. That's working pretty well because the redirects are simple.

I need help though. I've written a short proposal on how I think the UX for
API docs and tutorials should work. I'm not trying a major jump here. This
is a small amount of change for a better experience.
For example, right now, if you're browsing API docs for 1.3.0 and want to
see the master version, you would hit the dropdown and select master. Then
you get taken to the home page for master and have to go find the document
again. (It's always done this.) It should stay on that document, but give
the latest version. Or for tutorials, if you request a really old, likely
broken tutorial because it happens to show up in google search results, the
site should kindly escort you to the latest, tested tutorial.

mod_rewrite and .htaccess can do this, but it requires regex skills that I
lack. I've tried hundreds of variations now, and would like some guidance.

Here's the proposal and my notes:
https://cwiki.apache.org/confluence/display/MXNET/Version+calls+and+redirects+for+Tutorials+and+API+documents

I appreciate your help with this!

Cheers,
Aaron


Re: [DISCUSS] Build OSX builds in CI (possibly with TravisCI).

2018-09-12 Thread Aaron Markham
Is there any way to make it not show a red X failure in the GitHub UI when
TravisCI fails? I keep going back to check what flakey test failed this
time and realizing that Jenkins is still running and it was the "not
required" Travis fail. The green checkmark makes me happy and it's easier
to keep an eye on what's going on. If Travis times out a lot of the time,
then most of our PRs will look red/bad/sad when they're not.

What about no failure flag set, but add a label that Travis failed or
if we can't control the flag, auto-set labels for each Travis and Jenkins
pass/fail so we still get the benefit of at-a-glance status checks.

On Wed, Sep 12, 2018 at 6:04 AM Marco de Abreu
 wrote:

> Hello,
>
> Travis CI has successfully been enabled just now. This means you will now
> see a new status under your PR which is called
> "continuous-integration/travis-ci/pr".
>
> The job only compiles MXNet on Mac and currently does not run unit tests -
> we expect the overall execution duration to be around 6 minutes and thus
> faster than the full Jenkins pipeline. The status is set to "not required"
> which means that it does not block merging if that job fails since the
> pipeline is still in beta. But in general, it would be good if committers
> review the results in case the job shows a failure. Our last known state is
> that the pipeline works properly, but we will keep everybody up to date in
> case we get aware of any problems.
>
> The next step will be integration of Python CPU unit tests. There will be a
> separate email if we got an update on that manner.
>
> Special thanks to Kellen Sunderland for the contribution of this Travis CI
> pipeline.
>
> Best regards,
> Marco
>
> On Wed, Sep 5, 2018 at 8:19 PM Tianqi Chen 
> wrote:
>
> > Alrite, then I think it is fine as long as we can kept up with build
> speed
> > without timeout.
> >
> >
> > Tianqi
> >
> > On Wed, Sep 5, 2018 at 9:14 AM kellen sunderland <
> > kellen.sunderl...@gmail.com> wrote:
> >
> > > Travis actually has explicit support for ccache, it's a platform
> feature.
> > > I've run it and it seems to work quite well.  See for example this
> build:
> > >
> https://travis-ci.org/KellenSunderland/incubator-mxnet/builds/424768656
> > >
> > > On Wed, Sep 5, 2018 at 7:10 PM Tianqi Chen 
> > > wrote:
> > >
> > > > Travis it self is stateless, which means ccache is not likely going
> to
> > > > work. As far as I understand, if jenkins master is in the public
> > domain,
> > > > you do not need to setup a vpn to the subset of the master.
> > > >
> > > > As for versions of MacOS, we are likely going to be fine with one
> > > version,
> > > > as usually the problems exhibits on mac are similar
> > > >
> > > > Tianqi
> > > > On Wed, Sep 5, 2018 at 9:04 AM kellen sunderland <
> > > > kellen.sunderl...@gmail.com> wrote:
> > > >
> > > > > @Tianqi: Yeah there's going to be a lot of trade-offs to using
> > > Travis.  I
> > > > > hope we can get it running fast enough with ccache that it won't
> > > timeout
> > > > > when running tests, but even that is questionable.  In my private
> > > testing
> > > > > it was running in about 35 minutes and the global timeout for
> Travis
> > > jobs
> > > > > is 45 minutes.  I'd say let's run it for a few builds and see how
> it
> > > > goes.
> > > > > It won't be enabled in a mode that blocks PRs any time soon.
> > > > >
> > > > > I don't think physical hardware is a great solution.  We would have
> > to
> > > > > purchase the hardware, then maintain security updates, install
> > > different
> > > > > versions of XCode / MacOS, setup a vpn to our jenkins master,
> etc.  I
> > > > would
> > > > > also worry that if the machine goes down for whatever reason it
> would
> > > > block
> > > > > PRs, and someone would have to be physically present to turn it
> back
> > > on.
> > > > > Even assuming we set all the hardware up it's still not scalable so
> > > we'd
> > > > > have to over-provision.
> > > > >
> > > > > I'm hoping the Travis solution works for the time being. If it
> > doesn't
> > > > > we'll have to take a look at a few other options, but I've spent a
> > fair
> > > > > amount of time thinking about this and I don't think there are any
> > good
> > > > > options that don't have trade-offs.
> > > > >
> > > > > @Lin: Great!  Thanks for the offer.  There'll be a few features we
> > want
> > > > to
> > > > > re-enable once the Job gets hooked up again.  I'll ping you when
> it's
> > > > ready
> > > > > and see if there's anything you think would be interesting to help
> > > with.
> > > > >
> > > > > -Kellen
> > > > >
> > > > > On Wed, Sep 5, 2018 at 6:58 PM Lin Yuan 
> wrote:
> > > > >
> > > > > > Hi Kellen,
> > > > > >
> > > > > > I would love to contribute. Please let me know if you have any
> > > > particular
> > > > > > work item that I can help.
> > > > > >
> > > > > > Best,
> > > > > >
> > > > > > Lin
> > > > > >
> > > > > > On Wed, Sep 5, 2018 at 9:51 AM Tianqi Chen <
> > tqc...@cs.washington.edu
> > > >
> > > > > > 

Re: [VOTE] Release MXNet version 1.3.0.RC0

2018-09-05 Thread Aaron Markham
0 (non-binding) If we have a problem that blocks users, and a solution in
hand... then we should fix it, but not at the expense of starting the
release cycle again just for one fix. Users can cherry pick or build from
master if they want the fix right away, right? I'd change my mind to -1 if
this wasn't the case, with good reason, and if the user impact was critical
to adoption or risks abandonment.


On Wed, Sep 5, 2018 at 9:57 AM Roshani Nagmote 
wrote:

> I believe everyone here is working hard to make MXNet a better framework
> for users. It's completely okay to have different opinions, we can decide
> together if this issue is a blocker or not after voting time is over.
>
> As I mentioned before, voting will end at 7 pm today. So there is still
> time to test the release. If there are any other issues anyone finds, I
> will be happy to start the process again and work on RC1. For now, I want
> to encourage everyone to utilize this time and vote. :)
>
> Thanks,
> Roshani
>
> On Tue, Sep 4, 2018 at 10:35 PM sandeep krishnamurthy <
> sandeep.krishn...@gmail.com> wrote:
>
> >1. As a Apache MXNet community member, I raised the concern of broken
> >functionality for the user. I explained and provided the data points
> on
> > the
> >issue, workaround and why I think it is important. If after all this,
> > you
> >think my vote is biased on my employer just because a user I quoted is
> > from
> >Amazon, this is more concerning to me on my voting abilities.
> >2. My -1 no where undermines the huge amount of effort that goes
> behind
> >the scene for a release to happen. Great respect and recognition for
> >everyone involved in all the releases of MXNet in the past and this. I
> >voted on my judgement of what may be good for the users of MXNet.
> >3. As pointed by Naveen & Chris, -1 are NOT veto. Feel free to decide
> >and progress on the release as we already have >3 +1 in this thread.
> >
> >
> > Best,
> >
> > Sandeep
> >
> > On Tue, Sep 4, 2018 at 8:29 PM Chris Olivier 
> > wrote:
> >
> > > btw, there are no vetoes on package releases:
> > >
> > > VOTES ON PACKAGE RELEASES
> > > 
> > >
> > > Votes on whether a package is ready to be released use majority
> approval
> > >  --
> > i.e.
> > > at least three PMC members must vote affirmatively for release, and
> there
> > > must be more positive than negative votes.Releases may not be vetoed.
> > > Generally
> > > the community will cancel the release vote if anyone identifies serious
> > > problems, but in most cases the ultimate decision, lies with the
> > individual
> > > serving as release manager. The specifics of the process may vary from
> > > project to project, but the 'minimum quorum of three +1 votes' rule is
> > > universal.
> > >
> > > On Tue, Sep 4, 2018 at 7:12 PM Sheng Zha  wrote:
> > >
> > > > Thanks for sharing your opinions, Thomas. Your recognition and
> respect
> > of
> > > > people's efforts on preparing the release candidate are certainly
> > > > appreciated.
> > > >
> > > > Now that the vote is set to fail thanks to the veto, there will be
> > plenty
> > > > of opportunities to include those bug fixes, including the one Zhi
> > > > mentioned [1], which was already merged in the master and yet chose
> not
> > > to
> > > > block this release with [2]. I will be happy to work with Roshani to
> > > > prepare another release candidate once ready.
> > > >
> > > > -sz
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/f02e952bec22c82cb00a6741390a78f55373311c97464997bb455a6c@%3Cdev.mxnet.apache.org%3E
> > > > [2]
> > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/85d3fcabb3437ba7f1af455cf69aa13eb3afd1ea1d1f6f891e9c339c@%3Cdev.mxnet.apache.org%3E
> > > >
> > > > On Tue, Sep 4, 2018 at 6:02 PM Thomas DELTEIL <
> > thomas.delte...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > -0
> > > > > (non-binding)
> > > > >
> > > > > If I may add some nuancing plus a personal data point as one of the
> > > users
> > > > > commenting in the bug report in question:
> > > > >
> > > > > - Performance vs. Basic functionality => I don't think high
> > performance
> > > > > use-cases and basic functionality are two obviously opposed
> concepts
> > > and
> > > > > see no contradiction in Hagay's and Sandeep's statements.
> > > > > Float16 support is feature of MXNet that provides more than twice
> the
> > > > > performance of Float32 on supported platforms, hence the high
> > > performance
> > > > > use-case. The bug is that the basic functionality of reloading a
> > saved
> > > > > float16 models is currently broken.
> > > > >
> > > > > - This bug vs Other bugs => Contrary the vast majority of the 140
> > open
> > > > bugs
> > > > > that are mentioned above, I would put to Sandeep's credit that this
> > one
> > > > bug
> > > > > has a PR open that 

Re: [LAZY VOTE] Consolidating developer guide in one place (cwiki preferred)

2018-09-04 Thread Aaron Markham
I'd like to call for a lazy vote on this before proceeding. Already had
some +1s but let's be sure.

The vote is to move developer guide info to cwiki. User guides would remain
on the website.

On Tue, Aug 21, 2018 at 12:53 PM sandeep krishnamurthy <
sandeep.krishn...@gmail.com> wrote:

> +1
> Thanks Lin and Aaron. I agree website to cover all user facing
> documentation and a separate consolidated and organized developer focussed
> docs in one place (cwiki).
>
>
> Note: Permissions on cwiki is currently not well managed with many people
> having full admin rights to edit/create/delete pages. Should be fine for
> now, but, when we start accumulating many documents and resources, we
> should probably revisit on Delete permissions.
>
>
> On Tue, Aug 21, 2018 at 11:57 AM Lin Yuan  wrote:
>
> > Hi Aaron,
> >
> > Thanks for your answer. I think it's a very worthwhile effort to move all
> > the developer related content from mxet.io website to a dedicated
> > developer
> > site. Would you like to initiate this effort?
> >
> > Best,
> >
> > Lin
> >
> > On Wed, Aug 15, 2018 at 3:47 PM Haibin Lin 
> > wrote:
> >
> > > +1
> > >
> > > On Wed, Aug 15, 2018 at 1:10 PM, Aaron Markham <
> > aaron.s.mark...@gmail.com>
> > > wrote:
> > >
> > > > Hi Lin, I agree with this organization. If you feel like somethings
> > > should
> > > > be transitioned from the website to the wiki, I can help with that,
> but
> > > for
> > > > the moment I've been suggesting that new developer-focused content be
> > > > placed on the wiki.
> > > >
> > > > On Tue, Aug 14, 2018 at 10:40 AM, Lin Yuan 
> > wrote:
> > > >
> > > > > Dear MXNet community,
> > > > >
> > > > > As a developer, I noticed we have some developer guide scattered in
> > > > > different websites (mxnet.io, cwiki):
> > > > >
> > > > > E.g.
> > > > >
> > > > > How to Create New Operators (Layers): [
> > > > > https://mxnet.incubator.apache.org/faq/new_op.html]
> > > > > A Guide to Implementing Sparse Operators in MXNet Backend [
> > > > > https://cwiki.apache.org/confluence/display/MXNET/A+
> > > > > Guide+to+Implementing+Sparse+Operators+in+MXNet+Backend
> > > > > ]
> > > > >
> > > > > When searching developer guide by keyword, only one of them can be
> > > > returned
> > > > > on either site.
> > > > >
> > > > > It will be more convenient for developers if all the developer
> guide
> > > > > resides on cwiki and all user guide (non-developer) on the
> mxnet.io
> > > > > website. We can add a link on mxnet.io to refer all developers to
> > > cwiki
> > > > > for
> > > > > guidance.
> > > > >
> > > > > Any comment is appreciated.
> > > > >
> > > > > Best Regards,
> > > > >
> > > > > Lin
> > > > >
> > > >
> > >
> >
>
>
> --
> Sandeep Krishnamurthy
>


Re: build from source instructions

2018-08-28 Thread Aaron Markham
Here's a PR that addresses this, and more, for review:
https://github.com/apache/incubator-mxnet/pull/12388


On Tue, Aug 28, 2018, 13:21 Lin Yuan  wrote:

> When a user chooses to build from source, it is reasonable to infer that
> they want to run the make process and install the python package
> subsequently. The current automated build script is confusing in that I
> really have no idea what I should do if I want to change some of the source
> code in MXNet. Furthermore, building from source on MacOS should have the
> same/similar process as in building from source on Linux since they have
> the same shell environment. Having two different build instructions on
> MacOS just adds additional confusion.
>
>
>
> On Tue, Aug 28, 2018 at 10:44 AM Bhavin Thaker 
> wrote:
>
> > The automated build script on macOS was written with the intention to
> have
> > an automated, easy and quick way to build and install MXNet by any user,
> > new-bie or advanced. The build script aims to provide repeatability and
> an
> > easy way to test the build instructions.
> >
> > Without the script, the build instructions had many combinations of
> > possibilities which would break for various users and there was no easy
> way
> > to test all the combinations.
> >
> > I propose that we have both well-written build instructions with
> > corresponding automated build script to ensure that the build
> instructions
> > are well-tested.
> >
> > Please remember that there can be multiple use-cases and user preferences
> > to build MXNet.
> >
> > Bhavin Thaker.
> >
> > On Tue, Aug 28, 2018 at 10:29 AM Afrooze, Sina 
> wrote:
> >
> > > +1 on fully automated scripts being more confusing than helpful. It's
> > > difficult to debug any issues when the entire instruction is to run a
> > > single script. - Sina
> > >
> > >
> > >
> > > On 8/28/18, 9:46 AM, "Lin Yuan"  wrote:
> > >
> > > Aaron,
> > >
> > > I agree the installation page is very confusing to me. When I first
> > > tried
> > > to build MXNet from source on MacOS, I was totally confused about
> the
> > > instruction. Why was it vastly different from building from source
> on
> > > Linux
> > > given these two OS have similar shell commands. I feel the
> automatic
> > > scripts on MacOS platform is rather confusing than simplifying.
> > >
> > > Lin
> > >
> > > On Mon, Aug 27, 2018 at 9:21 PM Steffen Rochel <
> > > steffenroc...@gmail.com>
> > > wrote:
> > >
> > > > Aaron - we should keep instructions how to build from source.
> > > Updating and
> > > > re-organizing makes sense to me.
> > > > Steffen
> > > >
> > > > On Mon, Aug 27, 2018 at 4:54 PM Aaron Markham <
> > > aaron.s.mark...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hello,
> > > > > I was looking into the C++ instructions and came across this
> > > seemingly
> > > > > pretty old page:
> > > > > https://mxnet.incubator.apache.org/install/build_from_source
> > > > >
> > > > > I think it has several inaccuracies as different/updated
> > > installation
> > > > info
> > > > > has been added to different pages.
> > > > >
> > > > > Should it be deleted?
> > > > >
> > > > > Or should a specific build from source page be maintained
> > > (moving/copying
> > > > > info from the other more recently updated pages)?
> > > > >
> > > > > I'm really thinking that it would be easier to maintain if each
> > OS
> > > had
> > > > its
> > > > > own page, Python/pip info had its own page, then bindings had
> > > their own
> > > > > pages.
> > > > >
> > > > > Other suggestions?
> > > > >
> > > > > Cheers,
> > > > > Aaron
> > > > >
> > > >
> > >
> > >
> > >
> > >
> >
>


build from source instructions

2018-08-27 Thread Aaron Markham
Hello,
I was looking into the C++ instructions and came across this seemingly
pretty old page:
https://mxnet.incubator.apache.org/install/build_from_source

I think it has several inaccuracies as different/updated installation info
has been added to different pages.

Should it be deleted?

Or should a specific build from source page be maintained (moving/copying
info from the other more recently updated pages)?

I'm really thinking that it would be easier to maintain if each OS had its
own page, Python/pip info had its own page, then bindings had their own
pages.

Other suggestions?

Cheers,
Aaron


Re: Consolidating developer guide in one place (cwiki preferred)

2018-08-15 Thread Aaron Markham
Hi Lin, I agree with this organization. If you feel like somethings should
be transitioned from the website to the wiki, I can help with that, but for
the moment I've been suggesting that new developer-focused content be
placed on the wiki.

On Tue, Aug 14, 2018 at 10:40 AM, Lin Yuan  wrote:

> Dear MXNet community,
>
> As a developer, I noticed we have some developer guide scattered in
> different websites (mxnet.io, cwiki):
>
> E.g.
>
> How to Create New Operators (Layers): [
> https://mxnet.incubator.apache.org/faq/new_op.html]
> A Guide to Implementing Sparse Operators in MXNet Backend [
> https://cwiki.apache.org/confluence/display/MXNET/A+
> Guide+to+Implementing+Sparse+Operators+in+MXNet+Backend
> ]
>
> When searching developer guide by keyword, only one of them can be returned
> on either site.
>
> It will be more convenient for developers if all the developer guide
> resides on cwiki and all user guide (non-developer) on the mxnet.io
> website. We can add a link on mxnet.io to refer all developers to cwiki
> for
> guidance.
>
> Any comment is appreciated.
>
> Best Regards,
>
> Lin
>


social media promotion / menu

2018-08-09 Thread Aaron Markham
Hi dev,

I've been asked a variety of times how we can improve social media
awareness on the site. We kind of had a social section, but it was really
only evident on the contribute page. Sharing any page now results in a
decent preview image.

To help raise awareness of our different media channels, I created a
"social" navbar. I have to say that pixel pushing is my least favorite
thing to do and while the site can use a redesign, I'd like to cover
functionality/requirements, then ask for design help if you all absolutely
hate it. Or if you like it, then great. Though, I think with design help it
would be even better! Let me know what you all think.

Should it be on every page? Adjust the button color? Make them smaller?
Don't use them, just use a text menu. Check it on your mobile to see how it
is implemented there in text only.

You can preview it here:
http://34.201.8.176/versions/social_media_update/
and I fixed up the social media section here too:
http://34.201.8.176/versions/social_media_update/community/contribute.html#social-media

PR: https://github.com/apache/incubator-mxnet/pull/12102

Cheers,
Aaron


Re: Significant windows build improvements from source

2018-08-04 Thread Aaron Markham
I followed the directions and got stuck on OpenBLAS. Then I tried to just
use MKL, but then mshadow build threw an error saying it required OpenBLAS.
#11769
 That whole part can really use some clarification.

On Sat, Aug 4, 2018, 05:21 Pedro Larroy 
wrote:

> Hi Qing
>
> The problem you are facing is while building opencv?
>
> We would need to automate dependency download and building as well as part
> of an improved windows experience. Right now I only automated the build
> steps assuming you have a working opencv and openblas build. This would be
> the next step and if someone has cycles to contribute I would be happy to
> review and assist.
>
> To summarize, yes the final goal is to have a fully automated setup that
> requires one or at most a couple of commands / steps to get MXNet up and
> running from sources in windows. If anyone is interested in helping with
> this, let's coordinate here.
>
> Pedro
>
> On Fri, Aug 3, 2018 at 9:09 AM Qing Lan  wrote:
>
> > Just an update from previous thread.
> >
> > I tried Windows build using
> > http://mxnet.incubator.apache.org/install/windows_setup.html
> instruction.
> > The OpenCV links there only contains prebuilt vs12 (VS 2013). Apart from
> > that, user need to create a new C++ project after VS2015 installation
> > because it did not trigger the c compiler installation. I copied the
> > command from Pedro's Python script and it improves a little. However,
> even
> > with these changes, I am still facing build failure. I think developers
> may
> > need some script to automate these process to build backend dependencies
> on
> > Windows. Do we have them already?
> >
> > Thanks,
> > Qing
> >
> > On 8/2/18, 10:10 AM, "Qing Lan"  wrote:
> >
> > Hi Pedro,
> >
> > Great works! Thanks for supporting Windows build in here.
> >
> > Thanks,
> > Qing
> >
> > On 8/2/18, 10:08 AM, "Marco de Abreu"  .INVALID>
> > wrote:
> >
> > Hello Pedro,
> >
> > These are great efforts! It's great to see that we are improving
> > our
> > situation around Windows.
> >
> > Looking forward to using it.
> >
> > Best regards,
> > Marco
> >
> > Pedro Larroy  schrieb am Do., 2.
> > Aug. 2018,
> > 19:04:
> >
> > > Hi
> > >
> > > I have taken some effort to cleanup the Windows build process
> > and condense
> > > it in a single python script, provided that the required
> > dependencies are
> > > installed (opencv, openblas and cuda/cudnn in gpu).
> > >
> > > Now there's a single command necessary to get a build:
> > >
> > > python ci\build_windows.py
> > >
> > > The build flavour can be given with the '-f' argument.
> > >
> > > This produces a windows_package.7z file with MXNet libraries,
> > python
> > > bindings and includes.
> > >
> > > For testing, just running a powershell script like:
> > >
> > >ci/windows/test_py3_cpu.ps1
> > >
> > > Will execute the unit tests.
> > >
> > > Let me know what you think.  I pretty much would like to see
> > these changes
> > > in the release branch which will help with any windows issues
> > that we might
> > > find.
> > >
> > > https://github.com/apache/incubator-mxnet/pull/11947
> > >
> > > Simple installation of dependencies and update to the
> > documentation about
> > > windows builds from source will come in the future.
> > >
> > > Pedro.
> > >
> >
> >
> >
> >
> >
>


Re: custom 404 and other error landing pages and server logs

2018-08-02 Thread Aaron Markham
Pedro, that would be cool.

On a related note, I've reviewed the results from the broken link checker
job and submitted a PR for 3 links out of >50 reports.
35 of these are in a "regression" category. The others are mostly
redirects, and the tool will get updated to deal with those (not report
temp moves, warn about perm moves).

Regarding the regression category, I propose that we remove this check in
favor of getting the server logs and analyzing issues from that end.

The 35 "broken links" are pages that existed at one time, but do not now.
There's no data if any site visitor even tried to hit these old pages. Yet,
this regression info comes in a report mixed with actual broken links and
make it seem like there's a problem when, well, maybe there's not. If we
had data to say, 500 users a week go to this link and we give them a 404.
Ok, that's a problem. But this regression check doesn't do that. It just
says "a link existed one day, and it isn't there now, and I (the link
checker) might be the only one in the world that cares." Also, this
regression check has no idea if users are going to a page that doesn't
exist due to a bad link in a blog or other outside resource. If we get the
server logs, we'll know what pages users are going to that aren't there. We
can make an informed decision for fixing it by redirecting it or by putting
something there and how to prioritize. Also, these 35 links just say
they're missing. Someone would have to investigate or just guess where to
redirect them, and maybe all that work doesn't even yield any benefit.

To summarize (remove the regression check, and...):
* check server logs on what pages users are getting errors on, and make
plans to fix those
* show users better error pages
* report broken links only when there's a live link on the site that yields
a true 404 (not a 301 or 303 or anything else)

Cheers,
Aaron


On Thu, Aug 2, 2018 at 3:17 PM, Pedro Larroy 
wrote:

> Yes, we can do something fun for these errors, something maybe with a cat
> an a DL theme, or some funny style transfer stuff.
>
> Pedro
>
> On Thu, Aug 2, 2018 at 11:54 PM Aaron Markham 
> wrote:
>
> > Hi everyone,
> > I would like to suggest that we adopt a custom landing page for missing
> > pages, rather than the dead-end error the website has now. Typically you
> > set this up by modifying the server config and pointing it to a custom
> > page.
> >
> > I'd like to know how we go about requesting that kind of change on the
> > Apache infra.
> >
> > Also, so that we know if people are getting 404s on certain pages, or if
> > pages are breaking and throwing 500 errors, we could look at the server
> > logs for the domain. A cron job that pushes them to s3 would work if we
> > can't get direct access.
> >
> > How do we get regular access to or a feed of these logs that are residing
> > on the web server?
> >
> > Cheers,
> > Aaron
> >
>


custom 404 and other error landing pages and server logs

2018-08-02 Thread Aaron Markham
Hi everyone,
I would like to suggest that we adopt a custom landing page for missing
pages, rather than the dead-end error the website has now. Typically you
set this up by modifying the server config and pointing it to a custom page.

I'd like to know how we go about requesting that kind of change on the
Apache infra.

Also, so that we know if people are getting 404s on certain pages, or if
pages are breaking and throwing 500 errors, we could look at the server
logs for the domain. A cron job that pushes them to s3 would work if we
can't get direct access.

How do we get regular access to or a feed of these logs that are residing
on the web server?

Cheers,
Aaron


Re: ApacheCon is just 60 days away. Are you ready?

2018-07-26 Thread Aaron Markham
Floating a couple ideas about this...
1) could add it to the home page in the space to the right of "learn more".

2) could add it to a news page or events page (after creating said page)

3) should definitely add it to the meetups and events page in the wiki

On Jul 26, 2018 10:16, "Naveen Swamy"  wrote:

Hi All, ApacheCon Organizers are requesting to spread the word about
ApacheCon. What do you guys think of adding the banner(temporarily) on
Apache MXNet website?

-- Forwarded message --
From: Rich Bowen 
Date: Thu, Jul 26, 2018 at 6:00 AM
Subject: ApacheCon is just 60 days away. Are you ready?
To: nsw...@apache.org


A few things you need to do before you pack your bags
View this email in your browser
<
https://mailchi.mp/0e024bf0b8da/apachecon-is-just-60-days-away-are-you-ready?e=a7dd8ac6e6
>
ApacheCon is just 60 days away!

Thanks again for speaking at ApacheCon. This event can't happen without
you, and we're enormously grateful for your hard work and preparation.

We need some help in these last two months leading up to the event. You are
the most effective spokesperson for the event - people are coming because
of you.

Here's what you can do to help spread the word.

* Follow us on Twitter - @apachecon
<
https://apachecon.us17.list-manage.com/track/click?u=40e225eb9c5953fb65d73f7bd=cc26150dad=a7dd8ac6e6
>
- and amplify the messages that we're sending there.

* Tell your friends, coworkers, and, most importantly, the people involved
with the Apache projects you're speaking about (ie, your dev@ and users@
mailing lists), that you'll be speaking, and encourage them to come.

* Put one (or more!) of our ad banners on your website, and link it to
http://apachecon.com/acna18
<
https://apachecon.us17.list-manage.com/track/click?u=40e225eb9c5953fb65d73f7bd=5ff3820cd9=a7dd8ac6e6
>
. You can find those banners here: http://www.apachecon.com/acna18/banners/
<
https://apachecon.us17.list-manage.com/track/click?u=40e225eb9c5953fb65d73f7bd=86cfcc5a6a=a7dd8ac6e6
>
These are suitable for your personal site, your company sites, and your
project sites.

* Register! Yes, I know, most of you have registered, but a few of you
still haven't. Use the speaker registration code we sent you earlier, or
contact us - h...@apachecon.com - if you've lost it. Register here
<
https://apachecon.us17.list-manage.com/track/click?u=40e225eb9c5953fb65d73f7bd=d3c9557e38=a7dd8ac6e6
>
.

* Book your hotel room. The hotel block closes on August 24th. Details
here: http://apachecon.com/acna18/venue.html
<
https://apachecon.us17.list-manage.com/track/click?u=40e225eb9c5953fb65d73f7bd=ec041c990e=a7dd8ac6e6
>


Finally, I know a few of you have, unfortunately, had to drop your speaking
slots. If that's you, I apologize for sending you this message - although
we would still appreciate any promotion you can do on our behalf. But
please see the unsubscribe link at the bottom if you're not interested in
hearing from us any more.

Thanks again, and we'll see you in Montreal!

Rich, for the ApacheCon Planners.
<
https://apachecon.us17.list-manage.com/track/click?u=40e225eb9c5953fb65d73f7bd=1831ad868f=a7dd8ac6e6
>
<
https://apachecon.us17.list-manage.com/track/click?u=40e225eb9c5953fb65d73f7bd=6aa75aa21b=a7dd8ac6e6
>
<
https://apachecon.us17.list-manage.com/track/click?u=40e225eb9c5953fb65d73f7bd=89b0fe9629=a7dd8ac6e6
>
*Copyright © 2018 The Apache Software Foundation, All rights reserved.*
You're on my list of speakers for ApacheCon North America 2018

*Our mailing address is:*
The Apache Software Foundation
3864 Grassy Creek Dr
<
https://maps.google.com/?q=3864+Grassy+Creek+Dr+Lexington+,++KY+40514=gmail=g
>
Lexington
<
https://maps.google.com/?q=3864+Grassy+Creek+Dr+Lexington+,++KY+40514=gmail=g
>,
<
https://maps.google.com/?q=3864+Grassy+Creek+Dr+Lexington+,++KY+40514=gmail=g
>
KY
<
https://maps.google.com/?q=3864+Grassy+Creek+Dr+Lexington+,++KY+40514=gmail=g
>
40514
<
https://maps.google.com/?q=3864+Grassy+Creek+Dr+Lexington+,++KY+40514=gmail=g
>
-1059

Add us to your address book



Want to change how you receive these emails?
You can update your preferences
<
https://apachecon.us17.list-manage.com/profile?u=40e225eb9c5953fb65d73f7bd=79c3a26c13=a7dd8ac6e6
>
or unsubscribe from this list
<
https://apachecon.us17.list-manage.com/unsubscribe?u=40e225eb9c5953fb65d73f7bd=79c3a26c13=a7dd8ac6e6=c7faa91e41
>.


[image: Email Marketing Powered by MailChimp]
<
http://www.mailchimp.com/monkey-rewards/?utm_source=freemium_newsletter_medium=email_campaign=monkey_rewards=40e225eb9c5953fb65d73f7bd=1
>


Re: [VOTE] Subscribe dev@ to Github Activities

2018-07-17 Thread Aaron Markham
+1, I don't read your emails anyways. Just kidding. I think it would be
good to see the action, even if I eventually have to setup filters if it
gets overwhelming.

On Tue, Jul 17, 2018 at 9:15 AM, Tianqi Chen 
wrote:

> +1, most of issue and PR activities are about development, and they belong
> to dev. It also helps us to recognizes contributors who are actively
> contributing but less vocal via emails -- there are many of them.
>
> Tianqi
>
> On Tue, Jul 17, 2018 at 8:47 AM, Anirudh  wrote:
>
> > -1
> >
> > The low signal to noise ratio would mean that we may miss important
> emails.
> > Even with the different filters that we may setup for dev@, the emails
> > would be too many to not miss the important ones. We would see more and
> > more people starting a design discussion on an issue or PR. Because of
> the
> > low signal to noise ratio on the dev@ list, many may miss these
> > discussions.
> >
> > Slowly, this would erode the purpose of the dev@ list as this means that
> > you don't really have to do anything explicitly on the dev@ list.
> > You can start a design discussion on a github issue. You can start a
> > vote/discussion on a github issue.
> >
> > Anirudh
> >
> > On Mon, Jul 16, 2018 at 4:35 AM, Timur Shenkao 
> wrote:
> >
> > > +1 if my vote can be taken into account
> > >
> > > On Mon, Jul 16, 2018 at 4:32 AM, Sheng Zha  wrote:
> > >
> > > > Hi,
> > > >
> > > > I'm starting a vote on subscribing dev@ to Github activities. See
> > > previous
> > > > discussion thread here
> > > >  > > > bfd01f0b78d3cb44034f566442@%3Cdev.mxnet.apache.org%3E>
> > > > .
> > > >
> > > > The vote lasts for three days and ends on 7/18/2018 at 9pm pst.
> > > >
> > > > -sz
> > > >
> > >
> >
>


Re: Squash/Merge PRs

2018-07-13 Thread Aaron Markham
> > In general, I think we should gradually start taking this into
> > > account
> > > > > when
> > > > > > reviewing with the goal of all PRs having a proper commit
> history.
> > > This
> > > > > > would allow us in the long term to completely move away from
> > > squashing.
> > > > > >
> > > > > > -Marco
> > > > > >
> > > > > > Pedro Larroy  schrieb am Fr., 13.
> Juli
> > > > > 2018,
> > > > > > 00:07:
> > > > > >
> > > > > > > This is a great discussion, thanks for raising the question
> Naveen.
> > > > > > >
> > > > > > > From my experience working in all kinds of software project of
> > > > varying
> > > > > > > sizes and flavours:
> > > > > > >
> > > > > > >1. People should be aware of git rebase interactive and have
> > > more
> > > > > > >hygiene in their PRs. If a PR is hygienic and is separated
> in a
> > > > set
> > > > > of
> > > > > > >semantic commits, it's better not squashed as it helps
> finding
> > > > bugs
> > > > > /
> > > > > > >bisecting. A "dirty" PR with WIP commits, is better
> squashed, or
> > > > > > > requested
> > > > > > >to rebase interactively on CR.
> > > > > > >2. Mixing different semantic changes on a single PR is an
> > > > > > anti-pattern,
> > > > > > >for example fixing whitespace or reformatting many lines and
> > > > > changing
> > > > > > > some
> > > > > > >code which is hidden in a bunch of trivial changes and could
> > > > cause a
> > > > > > > bug or
> > > > > > >major change of behaviour.
> > > > > > >3. Agreed what with others have mentioned about small
> > > incremental
> > > > > > steps
> > > > > > >better than huge PRs. We also have JIRA or issues to relate
> a
> > > set
> > > > of
> > > > > > PRs
> > > > > > >together. If not possible, PR with a set of non squashed
> commits
> > > > is
> > > > > > also
> > > > > > >good.
> > > > > > >
> > > > > > > Hope it helps.
> > > > > > >
> > > > > > > Pedro.
> > > > > > >
> > > > > > > On Thu, Jul 12, 2018 at 11:47 PM Naveen Swamy <
> mnnav...@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > @Aaron, I do not think most contributors(SDE or not) were
> even
> > > > aware
> > > > > > that
> > > > > > > > their commits are getting squashed into one and thereby
> others
> > > > losing
> > > > > > > > credit on that work. I would assume no bad intentions there.
> > > > > > > >
> > > > > > > > @Hao,
> > > > > > > > Agree to breaking down into small and reasonable sized PRs,
> but I
> > > > > think
> > > > > > > > very small PRs will overwhelm the CI as it stands since it
> runs
> > > > > > > > everything(this is a separate topic and needs fixing).
> > > > > > > >
> > > > > > > > For cases similar to Aaron's he can raise the PR for the doc
> > > > > > > > work(regardless of fork or not) and add other contributors as
> > > > > > co-authors.
> > > > > > > >
> > > > > > > > @Marco, co-author might not be suitable always suitable and
> agree
> > > > > with
> > > > > > > Hao
> > > > > > > > that should be used on exceptions. co-author also makes it
> hard
> > > to
> > > > > see
> > > > > > > the
> > > > > > > > contributions individually.
> > > > > > > >
> > > > > > > > @Marco, why can we not have Rebase merge enabled on the repo
> and
> > > > use
> > > > > > that
> > > > > > > > whe

Re: Squash/Merge PRs

2018-07-12 Thread Aaron Markham
My point was about collaboration, or the lack thereof, and how the tooling
and apparent rewards from contribution statistics can reinforce lone ranger
behavior. People can and should be proud of their work, but why does it
have to be alone? Working within the context of a team is something to be
proud of too, but if your stats and standing are graded by how the commits
and merges land, and counting lines of code, then incentives of the system
are skewed.
How do we go about implementing the co-author process? It sounds like
something worth doing!

On Thu, Jul 12, 2018 at 11:15 AM, Hao Jin  wrote:

> Hi Aaron,
> To be fair, this discussion has nothing to do with "pride" of SDEs, but
> rather a discussion on what is a better software engineering practice for
> the project. Breaking a large project into smaller tasks is a good software
> engineering practice. This article(https://arxiv.org/pdf/1702.01715.pdf)
> on
> Google's software engineering practice says: "Engineers are encouraged to
> keep each individual change small​, with larger changes preferably broken
> into a series of smaller changes that a reviewer can easily review in one
> go.". If you have concerns or your comments on this practice, we can take
> the discussion offline. On the other hand, an important spirit of Apache
> community is that "one must interact with others, and share vision and
> knowledge"(https://community.apache.org/contributors/), if you observed
> that a majority of the contributors have serious problems with their
> writing maybe you can share some tips and hints on how to write better
> documentations, in that way not only a lot of us within the community can
> have some growth but also you can save some time for writing more good
> documentations and blogposts for MXNet, don't you think so? Or, if you only
> have to fix the doc for someone once in a while, I definitely agree that
> you should be given the credit for that, and that's where the co-author
> field can be useful, which was exactly what I was saying in my previous
> email. Thanks!
> Hao
>
> On Thu, Jul 12, 2018 at 8:16 AM, Aaron Markham 
> wrote:
>
> >  This is a great discussion, and close to my heart, because I want to
> > include more writers and editors in our community, and I'm struggling to
> > see how to best manage this. It seems like being the sole contributor on
> a
> > feature is like an engineer's Lone Ranger badge of pride! I feel like it
> > should be a red flag.
> >
> > I work in collaboration with dozens of engineers to improve their
> > documentation, explain their features, change flows to improve user
> > experience, and sometimes fix bugs and write code. When the PR's docs has
> > great coverage, is clear, and not loaded with bad grammar and spelling
> > mistakes, I will put comments in a review. But sometimes there needs to
> be
> > a complete rework such that hundreds of review comments isn't feasible,
> and
> > they can't be properly addressed. In a lot of these cases, I commit my
> > changes to someone else's fork. Now I know the people I work with on
> their
> > PRs appreciate my help, but when all of this work gets squashed down to
> one
> > commit, I'm pretty much regularly getting squashed out. I'm not sure who
> > else is in this boat, but does the community recognize our contributions
> > when this is the general mode of operation?
> >
> > Regarding co-author - I'm not sure how people would feel about me being a
> > co-author on their work. Documentation and clarity in your work product
> is
> > important, but people's views on their personal *code* contribution seems
> > to be more important than the overall code & feature quality itself when
> > docs are part of the package. I feel that if I do follow-on PRs instead
> of
> > fixing things before they are merged, that we would be releasing
> incorrect,
> > incomplete, and inferior product. But, in absence of a better solution,
> > maybe that's the pill we have to swallow.
> >
> > -1 on lots of small PRs (until we expand our range of committers and
> reduce
> > the latency in reviews and merges).
> > +1 on improving the quality of commit messages, so we don't have to
> squash
> > & merge all of the time.
> > +1 on improving the practice of better commit & merge management so that
> > the commit history is concise and meaningful. (I'm guilty of this, and
> > certainly need to improve here.)
> > +1 on co-author - assuming that's what most everyone thinks is a good
> > solution for now. I'm unclear on how this gets managed though.
> >
> > Cheers,
> > Aaron
> >
> >
> > On Thu

Re: Squash/Merge PRs

2018-07-12 Thread Aaron Markham
 This is a great discussion, and close to my heart, because I want to
include more writers and editors in our community, and I'm struggling to
see how to best manage this. It seems like being the sole contributor on a
feature is like an engineer's Lone Ranger badge of pride! I feel like it
should be a red flag.

I work in collaboration with dozens of engineers to improve their
documentation, explain their features, change flows to improve user
experience, and sometimes fix bugs and write code. When the PR's docs has
great coverage, is clear, and not loaded with bad grammar and spelling
mistakes, I will put comments in a review. But sometimes there needs to be
a complete rework such that hundreds of review comments isn't feasible, and
they can't be properly addressed. In a lot of these cases, I commit my
changes to someone else's fork. Now I know the people I work with on their
PRs appreciate my help, but when all of this work gets squashed down to one
commit, I'm pretty much regularly getting squashed out. I'm not sure who
else is in this boat, but does the community recognize our contributions
when this is the general mode of operation?

Regarding co-author - I'm not sure how people would feel about me being a
co-author on their work. Documentation and clarity in your work product is
important, but people's views on their personal *code* contribution seems
to be more important than the overall code & feature quality itself when
docs are part of the package. I feel that if I do follow-on PRs instead of
fixing things before they are merged, that we would be releasing incorrect,
incomplete, and inferior product. But, in absence of a better solution,
maybe that's the pill we have to swallow.

-1 on lots of small PRs (until we expand our range of committers and reduce
the latency in reviews and merges).
+1 on improving the quality of commit messages, so we don't have to squash
& merge all of the time.
+1 on improving the practice of better commit & merge management so that
the commit history is concise and meaningful. (I'm guilty of this, and
certainly need to improve here.)
+1 on co-author - assuming that's what most everyone thinks is a good
solution for now. I'm unclear on how this gets managed though.

Cheers,
Aaron


On Thu, Jul 12, 2018 at 5:19 AM, Anton Chernov  wrote:

> Unfortunately there has been significant push back for small iterative PR's
> and as a result they have grown substantially involving multiple
> contributors, multiple commits, sometimes multiple branches to be worked
> on.
>
> I fully agree and support all points that Jin raised:
>
> 1) Most contributions should be broken down into smallest possible PR's.
> 2) If a PR is small enough a single person can complete it.
> 3) If a majority of PR is done by someone, and there's some minor issue
> he/she needs help with it could be done in a follow up PR by anybody,
> including the reviewer.
>
> Best regards,
> Anton
>
> чт, 12 июл. 2018 г. в 10:11, Hao Jin :
>
> > +1 for Marco's proposal on using the co-author field. IMHO the usage of
> > co-author field should not be that often, consider:
> > 1) If a PR is so big that it needs multiple people to contribute a
> > substantial part of it, why can't that PR be broken down into smaller PRs
> > each submitted by single one of them?
> > 2) If a PR is small enough (for example, < 300 lines), why can't a single
> > person complete it?
> > 3) If a majority of PR is done by someone, and there's some minor issue
> > he/she needs help with(a small bug, incomplete doc), why can't that be
> done
> > through code reviews?
> > From the above 3 situations we can see that collaborations on individual
> > PRs should not be quite often, but I do agree that it could still be
> > necessary when someone lacks the related expertise/knowledge on some
> > necessary part of that PR given that the required knowledge cannot be
> > picked up in a short period of time.
> >
> > I do agree that keeping the commit histories of PRs clean is very
> important
> > as it could be confusing when reviewing PRs, but that really depends on
> > personal preferences (For my case, I usually do "git commit --amend" on
> > trivial changes and get a new commit for bigger changes such as a
> > checkpoint of my whole PR). With growing the community and attracting
> more
> > contributors as a high priority for our project, I would prefer that we
> do
> > not put even more burden on the contributors by asking them to manage and
> > squash the commits themselves just for the not-so-often cases of having
> > multiple contributors on a single PR.
> > Best regards,
> > Hao
> >
> > On Thu, Jul 12, 2018 at 12:35 AM, Marco de Abreu <
> > marco.g.ab...@googlemail.com.invalid> wrote:
> >
> > > Hi Naveen,
> > >
> > > I'm in favour of the squashing, considering the number of commits in
> some
> > > PRs and especially because of some people making commit messages a la
> > "fix"
> > > "fix" "fix" all the time. Additionally, it gets hard (not impossible,

Re: Abandoned MXNet readthedocs site

2018-07-05 Thread Aaron Markham
If they could redirect to the official website that would be ideal.
Otherwise, we'd be supporting yet another doc outlet and possibly run into
the same problem of it getting out of sync and being incorrect.

On Jul 5, 2018 10:32, "Craig Russell"  wrote:

Peanut gallery please ignore if off topic.

Does the mxnet project want to manage the page or remove it?

Would it be better to improve the readthedocs page since it is so
frequently found when searching for mxnet install?

I assume Hen is trying to get the page owner changed to a (P)PMC managed
account?

Craig


> On Jul 5, 2018, at 10:22 AM, Hen  wrote:
>
> I’ve Twitter DM’d one of the core members of ReadTheDocs asking for help.
>
> On Thu, Jul 5, 2018 at 8:43 AM Chris Mattmann mailto:mattm...@apache.org>> wrote:
> Hi Hen happy 4th looks like a brand mgmt. topic to me…
>
>
>
> Cheers,
>
> Chris
>
>
>
>
>
>
>
>
>
> From: Hen mailto:bay...@apache.org>>
> Reply-To: >

> Date: Tuesday, July 3, 2018 at 10:44 AM
> To: mailto:dev@mxnet.incubator.apache.org
>>
> Cc: "legal-disc...@apache.org " <
legal-disc...@apache.org >

> Subject: Re: Abandoned MXNet readthedocs site
>
>
>
> Chris/Mark - is this a Legal (Chris) topic or a Brand Management (Mark)
>
> topic?
>
>
>
> I’m hesitant to use a Trademark C for what’s in effect a lost account,
>
> but that’s what readthedocs seem to be asking for.
>
>
>
> On Mon, Jun 25, 2018 at 3:15 PM Markham, Aaron

>
> wrote:
>
>
>
> Hi, just bumping this issue again. Can we get a trademark C sent to the
>
> readthedocs folks, so they'll go ahead and remove the derelict website?
>
> It's still the #4 result in Google for "mxnet install".
>
>
>
> What's Apache's process?
>
>
>
> Thanks,
>
> Aaron
>
>
>
> On 5/16/18, 8:20 AM, "Hen" mailto:bay...@apache.org>>
wrote:
>
>
>
>  It seems we (Apache MXNet) have an old documentation site posted at:
>
>
>
>  https://newdocs.readthedocs.io/en/latest/ <
https://newdocs.readthedocs.io/en/latest/>

>
>
>
>  and only the absent @Awyan has the permissions to the site. See
>
>  https://github.com/apache/incubator-mxnet/issues/10409 <
https://github.com/apache/incubator-mxnet/issues/10409> for more
>
> details.
>
>
>
>  Raising this with readthedocs.org , we've
been asked to submit a DMCA

>
>  request to take down the account:
>
>
>
>  https://github.com/rtfd/readthedocs.org/issues/4042 <
https://github.com/rtfd/readthedocs.org/issues/4042>

>
>
>
>  Given our docs are Apache licensed I suspect it's really a trademark
>
>  takedown (confused users etc). Does anyone here have any guidance
with
>
> this
>
>  situation? Has anyone had this issue with readthedocs before? Should
>
> we be
>
>  sending them a Trademark takedown request?
>
>
>
>  Thanks,
>
>
>
>  Hen
>
>
>
>
>
>
>
>
>

Craig L Russell
Secretary, Apache Software Foundation
c...@apache.org  http://db.apache.org/jdo <
http://db.apache.org/jdo>


Re: landing pages for the website

2018-07-02 Thread Aaron Markham
I've created a PR for an ecosystem page. Please let me know your feedback.

https://github.com/apache/incubator-mxnet/pull/11528


On Sat, Jun 23, 2018 at 2:50 AM, Ivan Serdyuk 
wrote:

> I wonder if there would be a page for a list of startups, which use your
> project (integration)? With details of type of cloud/inter-cloud (private;
> hybrid; big commercail ones)
>
> On Fri, Jun 22, 2018 at 9:44 PM, Steffen Rochel 
> wrote:
>
> > Hi Aaron - support Mu's suggestion to add an ecosystems page to add links
> > to MXNet related projects (gluon toolkits, mxnet model server, keras,
> onnx,
> > sockeye etc) to promote the solutions in the MXNet ecosystem. What do you
> > suggest as next steps?
> > Steffen
> >
> > On Mon, Jun 11, 2018 at 5:19 PM Mu Li  wrote:
> >
> > > Thanks for the proposal, Aaron. The landing page idea is great. I feel
> we
> > > should group MXNet topics into landing pages, such as Gluon you
> > mentioned,
> > > also performance, and deployment. But I'm less sure if we want to add
> too
> > > many contents from 3rd party projects into MXNet, otherwise, it may
> make
> > > the MXnet docs have too many dependencies and hard to maintain. We may
> > just
> > > create an "ecosystem" page to add links to all high-quality 3rd party
> > > libraries.
> > >
> > > On Mon, Jun 11, 2018 at 5:11 PM, Aaron Markham <
> > aaron.s.mark...@gmail.com>
> > > wrote:
> > >
> > > > Hi dev@,
> > > > We've had a lot of great expansions to MXNet's functionality and
> family
> > > of
> > > > tools in recent months. I would like to gauge the temp on what would
> be
> > > > good to highlight, and what would be good as navigation items with
> > actual
> > > > landing pages.
> > > >
> > > > For example, when ONNX was added, being in a contrib package, it was
> > > going
> > > > to totally buried, so it has a navigation item under Docs and a
> landing
> > > > page with some usage details. This page could certainly be enhanced,
> or
> > > > there could also be a landing page for it in the docs folder.
> > > > https://mxnet.incubator.apache.org/api/python/contrib/onnx.html
> > > >
> > > > Gluon needs a new landing page, and could probably be folded into the
> > > Docs
> > > > navigation. Why landing pages? We shouldn't have navigation items
> that
> > > take
> > > > you off of the site without warning. Github would be an exception.
> But
> > > for
> > > > books and tutorials suites that are off-deck, I think it would be a
> > good
> > > > habit to describe what it is and indicate where it goes. The
> tutorials
> > > page
> > > > has this notion with little icons now.
> > > >
> > > > The new Scala infer package will be buried once it rolls off of the
> new
> > > > highlights section of the home page. Maybe we should have navigation
> > > items
> > > > for things like model serving and inference along with landing pages
> > for
> > > > each?
> > > >
> > > > Then we have a couple of projects outside MXNet, but directly add
> > value,
> > > > and could be highlighted with a landing page to describe what it
> does,
> > > how
> > > > to install it, plus links to the projects and some examples:
> > > >
> > > > * MXNet Model Server
> > > > * Keras-MXNet
> > > >
> > > > I'm sure there's more. Let's discuss and promote all these good
> things.
> > > >
> > > > Cheers,
> > > > Aaron
> > > >
> > >
> >
>


Re: Request for Sign up

2018-06-15 Thread Aaron Markham
Hi,
Have you checked out this page?
https://mxnet.incubator.apache.org/community/contribute.html

The dev comm section has info on how to subscribe to the mail list.

I think Steffen already added you to slack.

On Fri, Jun 15, 2018, 16:33 Kalyanee Chendke  wrote:

> Hello,
>
> Bumping this up. I would like to request to sign up.
>
> -Kalyanee
>
> Kalyanee
>
> On Thu, Jun 7, 2018 at 11:27 AM, Kalyanee Chendke 
> wrote:
>
> > Hey!
> >
> > I would like to request for sign up to the MXNet Apache Mailing list &
> > also to the MXNet Slack Channel.
> >
> > Kalyanee
> >
>


Re: Update on 1.2.1 release

2018-06-13 Thread Aaron Markham
I'd keep an eye on this one...  Flaky test: test_gru_bidirectional #11219

https://github.com/apache/incubator-mxnet/issues/11219

Just reran several PR's CI runs that all had the same error!

On Wed, Jun 13, 2018 at 5:42 PM, Anirudh  wrote:

> Hi,
>
> PRs still in progress : #11127, #11236, #11210, #11054, #11216.
>
> We are currently facing two issues which are delaying the merge of some of
> these PRs:
> 1. Flaky tests for scala API. A PR is already out to disable the test:
> https://github.com/apache/incubator-mxnet/issues/11249
> 2. Builds breaking on windows:
> https://github.com/apache/incubator-mxnet/issues/11265
>
> Anirudh
>
>
> On Tue, Jun 12, 2018 at 11:59 AM, Anirudh  wrote:
>
> > Hi all,
> >
> > Here are the PRs that are being tracked for 1.2.1 release:
> >
> > Related to the save_params backwards incompatible change: #11127 (In
> > Progress), #11236 (In Progress), #11210 (In Progress)
> > MKLDNN Fixes: #11212 (In Progress)
> > Cross compilation for armv7 : #11054 (In Progress)
> > Scala Inference Memory leak fix: #11216 (In Progress)
> > Docs changes: #11211 (Merged)
> > Inplace RELU Activation, Slice operator perf improvement: #11142 (Merged)
> > Use cudnnv7 for depthwise conv #11233 (Merged)
> >
> > Please let me know if I have missed something.
> >
> > Anirudh
> >
>


Re: test/versions in the website

2018-06-13 Thread Aaron Markham
It's not on the repo.
https://github.com/apache/incubator-mxnet-site/tree/asf-site/test
Must be sitting on the server.
Maybe infra can take a look.


On Wed, Jun 13, 2018 at 2:38 PM, Naveen Swamy  wrote:

> I think this was some testing during migration to Apache infra, I think it
> probs is sitting on a branch on mxnet-site repo
>
> > On Jun 13, 2018, at 11:13 PM, Pedro Larroy 
> wrote:
> >
> > Hi
> >
> > We have this url alive, and google is indexing:
> > http://mxnet.incubator.apache.org/test/versions/
> >
> > Is this url right, or is it pointing some users to the wrong
> documentation?
> >
> > Pedro
>


landing pages for the website

2018-06-11 Thread Aaron Markham
Hi dev@,
We've had a lot of great expansions to MXNet's functionality and family of
tools in recent months. I would like to gauge the temp on what would be
good to highlight, and what would be good as navigation items with actual
landing pages.

For example, when ONNX was added, being in a contrib package, it was going
to totally buried, so it has a navigation item under Docs and a landing
page with some usage details. This page could certainly be enhanced, or
there could also be a landing page for it in the docs folder.
https://mxnet.incubator.apache.org/api/python/contrib/onnx.html

Gluon needs a new landing page, and could probably be folded into the Docs
navigation. Why landing pages? We shouldn't have navigation items that take
you off of the site without warning. Github would be an exception. But for
books and tutorials suites that are off-deck, I think it would be a good
habit to describe what it is and indicate where it goes. The tutorials page
has this notion with little icons now.

The new Scala infer package will be buried once it rolls off of the new
highlights section of the home page. Maybe we should have navigation items
for things like model serving and inference along with landing pages for
each?

Then we have a couple of projects outside MXNet, but directly add value,
and could be highlighted with a landing page to describe what it does, how
to install it, plus links to the projects and some examples:

* MXNet Model Server
* Keras-MXNet

I'm sure there's more. Let's discuss and promote all these good things.

Cheers,
Aaron


Re: Regarding 1.2.1 patch release

2018-06-08 Thread Aaron Markham
Hi Anirudh,
Marco brought my attention to CI failures on the 1.2.1 builds with regard
to docs. The failures are because the build scripts that are in the 1.2.0
branch are out of date and have problems due to package updates that are
incompatible with each other.

To fix this you'll need to bring all of the docs building scripts
up-to-date. There were a few updates prior to #10485, so if those are
missing from 1.2.0, they might need to be included too. (#10270,#10010,
#9878)

Please use this major update to docs building:
https://github.com/apache/incubator-mxnet/pull/10485

These update docs to reflect the changes:
https://github.com/apache/incubator-mxnet/pull/10728
https://github.com/apache/incubator-mxnet/pull/10785

This updates the Jenkinsfile to secure nodes:
https://github.com/apache/incubator-mxnet/pull/11066

Please let me know if you have any questions.

Cheers,
Aaron

On Fri, Jun 8, 2018 at 6:34 AM, Anton Chernov  wrote:

> I would like to propose [1] as an important fix for RaspberryPi's for the
> 1.2 patch release.
>
> -- Anton
>
> [1] https://github.com/apache/incubator-mxnet/pull/11054
>
> 2018-06-08 1:44 GMT+02:00 Zheng, Da :
>
> > Hello Anirudh,
> >
> > There is a test (test_hybrid_multi_context) for
> https://github.com/apache/
> > incubator-mxnet/pull/10706
> > It's also tested by the C++ unit tests in https://github.com/apache/
> > incubator-mxnet/pull/10979
> > https://github.com/apache/incubator-mxnet/pull/10979/files#diff-
> > 8118d4fd8d897a9177f48257a466ea13R435
> >
> > Best,
> > Da
> >
> > On 6/7/18, 3:37 PM, "Anirudh"  wrote:
> >
> > Hi Da,
> >
> > Can you please open a PR to the 1.2 branch with the following PRs
> > mentioned
> > by you and Tao cherry picked onto release branch.
> >
> > https://github.com/apache/incubator-mxnet/pull/10979
> > https://github.com/apache/incubator-mxnet/pull/10731
> > https://github.com/apache/incubator-mxnet/pull/10651
> > https://github.com/apache/incubator-mxnet/pull/10624
> > https://github.com/apache/incubator-mxnet/pull/10619
> > https://github.com/apache/incubator-mxnet/pull/10616
> > https://github.com/apache/incubator-mxnet/pull/10918
> > https://github.com/apache/incubator-mxnet/pull/10613
> >
> > I am a little concerned about the below two PRs since they don't have
> > enough tests:
> > https://github.com/apache/incubator-mxnet/pull/10706
> > https://github.com/apache/incubator-mxnet/pull/10810
> >
> > 
> > Can you please talk about the test coverage of these two PRs.
> >
> > Hi Tao,
> >
> > To answer your question about the criteria for choosing PRs for patch
> > release: I think we haven't done patch release many times before and
> we
> > don't have a clear set of criteria on what can go into the patch
> > release.
> > The main intention of doing patch release is to fix the undocumented
> > backwards incompatible change from 1.1. Along with this critical
> fixes
> > should also be pushed out. AFAIK, The "critical" here is not clearly
> > defined by the community yet and we should use our best judgement
> here.
> > According to me,  everything which has potential to impact a large
> > number
> > of MXNet users can be considered critical.
> >
> > The timeline for the patch release is "as soon as possible".
> > https://github.com/apache/incubator-mxnet/pull/11049 is not critical
> > and
> > will show up in mxnet.io docs in master when it is merged. I would
> be
> > little hesitant about https://github.com/apache/
> > incubator-mxnet/pull/11095
> > since it lacks test currently. We can consider
> > https://github.com/apache/incubator-mxnet/pull/11047 if it is merged
> > in
> > time.
> >
> > Anirudh
> >
> >
> >
> > On Thu, Jun 7, 2018 at 2:27 PM, Naveen Swamy 
> > wrote:
> >
> > > Hi Anirudh,
> > >
> > > I would like to get the fixes that was made to publish to Maven
> into
> > 1.2.1
> > > -- currently in PR https://github.com/apache/
> > incubator-mxnet/pull/11147/
> > >
> > > Also additionally, I would like to get the fix for
> > > https://github.com/apache/incubator-mxnet/issues/10436 - currently
> > another
> > > contributor Andrew is working on it.
> > >
> > > -Naveen
> > >
> > > On Thu, Jun 7, 2018 at 8:33 AM, Lv, Tao A 
> > wrote:
> > >
> > > > Thanks for bringing this up, Da!
> > > >
> > > > It would be great if we can have these fixes into 1.2.1 patch
> > release,
> > > > especially for https://github.com/apache/
> > incubator-mxnet/pull/10651, it
> > > > has fixed https://github.com/apache/incubator-mxnet/issues/11028
> .
> > > >
> > > > What I want to add are:
> > > > 1. https://github.com/apache/incubator-mxnet/pull/10810 which
> > fixed
> > > > https://github.com/apache/incubator-mxnet/issues/10809 .
> > > > 2. 

Re: Github link on MXNet Homepage

2018-06-07 Thread Aaron Markham
What about a GitHub icon instead the word?

Could also use a per page GitHub link to take you to the current page's
code on GitHub instead of just a link to the project home. I like how
docker does a similar thing to solicit feedback and contributions.

On Thu, Jun 7, 2018, 10:18 Marco de Abreu 
wrote:

> Yeah, why not :)
>
> -Marco
>
> On Thu, Jun 7, 2018 at 7:03 PM singh.raja...@gmail.com <
> singh.raja...@gmail.com> wrote:
>
> > Hi All,
> >
> > On MXNet home page, the link to github is currently inside
> > Community->GitHub. I think we should have link to GitHub readily
> available
> > on homepage ( one click), similar to other frameworks homepage out there.
> >
> > WDYT?
> >
> > Thanks
> > Rajan
> >
> >
>


Re: PR validation and runtime of CI

2018-06-07 Thread Aaron Markham
I'd like to disable flaky tests until they're fixed.
What would the process be for fixing a failure if the tests are done
nightly? Would the commit be reverted? Won't we end up in the same
situation with so many flaky tests?

I'd like to see if we can separate the test pipelines based on the content
of the commit. I think that md, html, and js updates should fly through and
not have to go through GPU tests.

Maybe some special flag added to the comment?
Is this possible?


On Wed, Jun 6, 2018 at 10:37 PM, Pedro Larroy 
wrote:

> Hi Team
>
> The time to validate a PR is growing, due to our number of supported
> platforms and increased time spent in testing and running models.  We are
> at approximately 3h for a full successful run.
>
> This is compounded with the failure rate of builds due to flaky tests of
> more than 50% which is a big drag in developer productivity if you can only
> get one or two CI runs to a change per day.
>
> I would want a turnaround time of less than 30 minutes and 0% failure rate
> on master.
>
> For this I propose working towards moving tests from CI to nightly,
> specially the ones that take most time or do black box testing with full
> training of models. And addressing flaky tests by either fixing them or
> disabling them.
>
> I would like to check if there's consensus on this previous plan so we are
> aligned on pursuing this common goal as a shared effort.
>
> Pedro.
>


Re: home page highlights section updates

2018-06-05 Thread Aaron Markham
I like that idea, Ivan. We certainly should have more meetups and workshops!


On Mon, Jun 4, 2018 at 12:41 PM, Ivan Serdyuk 
wrote:

> Greetings, Aaron. I wonder if your attempt monitor/track improvements would
> lead to easier schema of inviting speakers (workshops first-most), for
> various events.
>
>
> On Sat, Jun 2, 2018 at 12:57 AM, Aaron Markham 
> wrote:
>
> > Hi everyone, happy Friday,
> >
> > I've mocked up a couple of modifications for the highlights on the home
> > page.
> > The goal is to expose more news, more tutorials, and other interesting
> > things people have been working on. If you've contributed something
> > interesting and want to highlight it, you should be able to suggest an
> > update to the page.
> > It would be great to have variety and fresh content on the site.
> >
> > Here are a couple of ideas with screenshots that shouldn't be to hard to
> > implement:
> > https://cwiki.apache.org/confluence/display/MXNET/
> > Website+Interface+Updates
> >
> > You can reply with feedback here or make comments on the wiki! What are
> > your thoughts on:
> > 1) getting fresh content on the site
> > 2) highlighting contributors and their contributions
> > 3) flow on changes and what kind of approvals are expected for swapping
> out
> > highlights
> > 4) UX ideas that are not too challenging to implement or maintain and
> work
> > well with mobile
> >
> > Cheers,
> > Aaron
> >
>


home page highlights section updates

2018-06-01 Thread Aaron Markham
Hi everyone, happy Friday,

I've mocked up a couple of modifications for the highlights on the home
page.
The goal is to expose more news, more tutorials, and other interesting
things people have been working on. If you've contributed something
interesting and want to highlight it, you should be able to suggest an
update to the page.
It would be great to have variety and fresh content on the site.

Here are a couple of ideas with screenshots that shouldn't be to hard to
implement:
https://cwiki.apache.org/confluence/display/MXNET/Website+Interface+Updates

You can reply with feedback here or make comments on the wiki! What are
your thoughts on:
1) getting fresh content on the site
2) highlighting contributors and their contributions
3) flow on changes and what kind of approvals are expected for swapping out
highlights
4) UX ideas that are not too challenging to implement or maintain and work
well with mobile

Cheers,
Aaron


installation page UX update

2018-05-30 Thread Aaron Markham
Hi everyone,
@kpmurali (Krishnan) has worked on some of the initial feedback for the
install UX. He has this preview link:
http://54.210.6.225/install/index.html?version=v1.2.0=Windows=Python=GPU

It has a couple nice features:
* When you choose different install paths, it puts the selections in the
URL, so you can share that or use it elsewhere and it'll take the user
directly to the instructions you want them to see.
* I uses a very obvious version selector that users can see and use to
change the directions (for pip versions).

If you like it, please give him a thumbs up on his PR, or provide some
feedback.
https://github.com/apache/incubator-mxnet/pull/11069

Cheers,
Aaron


Good First Issue label

2018-05-29 Thread Aaron Markham
Just wanted to bring everyone's attention to the label: Good First Issue.

When you're going through new issues and labelling, keep this one in mind,
so there are opportunities for new members to the project. I think a lot of
people would like to contribute, but often the issues are so complex they
don't know where or how to start.

If you see a pattern in issues or a task that can be broken up into
manageable bits, create a Good First Issue - something that is clear and
attainable. I think the other similar label, Call for Contribution,
encompasses larger feature sets, rather than introductory kinds of
contributions.

Cheers,
Aaron


Re: Introduction of restricted slaves and jobs

2018-05-25 Thread Aaron Markham
Thanks Marco, this is a welcome improvement.


On Fri, May 25, 2018, 05:56 Marco de Abreu 
wrote:

> Hello MXNet community,
>
> I'd like to announce the launch of restricted slaves and jobs for our CI
> system.
>
> The purpose of this feature is to allow separating slaves that execute
> arbitrary code from verified code like on our version-branches and the
> master. This step is necessary in order to increase the security of
> produced artefacts.
>
> Until now, the generation of user-facing artefacts like our website was run
> on the same instances as unverified code from Pull Requests. This could
> potentially have been abused to deploy a virus on our slaves (although they
> are recycled very frequently through the auto scaling system) that alters
> the artefact generation processes and attaches malicious code to it.
>
> In order to mitigate this attack vector, we're introducing restricted
> slaves and jobs. From now on, any user-facing output like nightly builds or
> the website, will have to be generated on slaves that are only executing
> code that has been verified by our committers by merging the changes into
> one of our branches.
>
> I'd like to invite everybody to review the documentation at
> https://cwiki.apache.org/confluence/display/MXNET/Restricted+jobs+and+nodes
> .
> Considering this is a security feature, I'd especially love to hear
> critical input or any ideas that would allow to poke holes into my
> solution.
>
> Best regards,
> Marco
>


Re: MXNet issues labeling

2018-05-21 Thread Aaron Markham
AI obviously.

On Mon, May 21, 2018 at 5:01 PM, Anirudh  wrote:

> Hi Roshani,
>
> Good suggestion! How will the bot decide what labels to add ?
>
> Anirudh
>
> On Mon, May 21, 2018 at 4:39 PM, Naveen Swamy  wrote:
>
> > +1 for the proposal to triage issues, I think committers should also do
> > this exercise to understand the customer pain.
> >
> > I am also inclined to use a bot account like how Tensorflow and other
> repos
> > do it, https://github.com/googlebot.
> > https://github.com/tensorflow/tensorflow/pull/19445#event-1638027271 -->
> > This is auto-tagged by the bot, it would be cool if we could do that as
> > well.
> >
> >
> >
> > On Mon, May 21, 2018 at 4:25 PM, sandeep krishnamurthy <
> > sandeep.krishn...@gmail.com> wrote:
> >
> > > Thanks,
> > >
> > > Roshani for starting this thread.
> > >
> > > Yes, I think labeling the issues will help a lot in driving the
> attention
> > > of contributors to specific areas and make it easy for new contributors
> > to
> > > search and pick their contribution.
> > >
> > > I agree manually doing it all the time is not scalable and efficient.
> > Your
> > > proposal on bot script to auto-label, similar to the working of Jenkins
> > bot
> > > to re-test, re-build actions, will be very useful and effective.
> Hence, I
> > > am more inclined to your *option 1* to have a bot account to add
> labels.
> > >
> > > Best,
> > > Sandeep
> > >
> > > On Mon, May 21, 2018 at 4:16 PM, Roshani Nagmote <
> > > roshaninagmo...@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > Some of us here at Amazon as a part of our day job, are triaging
> Github
> > > > issues to find where MXNet users are experiencing difficulty and help
> > the
> > > > community focus on those areas. This is done by assigning labels to
> the
> > > > Github issues. We do know that only labeling won’t solve the real
> > problem
> > > > but we will expand our scope to also attempt to resolve the issues.
> > > > Categorizing issues could also help contributors and maintainers who
> > > know a
> > > > particular area to pick up the issue and help the user.
> > > >
> > > > Right now, we just manually go through the issues. If they are
> > questions,
> > > > we redirect users to start a discussion on discuss forum, find the
> > > > appropriate labels and then ask one of the committers to add those
> > > labels.
> > > > This process is not very smooth as its completely manual and every
> time
> > > we
> > > > need to ask committers to add labels.
> > > >
> > > > We want to be able to automate/simplify this issue labeling process.
> > > > Right now, as far as I know, there's no way for non-committers to add
> > > > labels. So, I want to propose two options:
> > > >
> > > > - Using a separate account having minimum permissions to run the bot
> > > script
> > > > which will do the labeling. For this, we will need an account to be
> > > created
> > > > from Apache infrastructure with proper access and they can control
> the
> > > > access for the account through
> > > > https://docs.aws.amazon.com/secretsmanager/latest/
> userguide/intro.html
> > > >
> > > > - Using one of the committers auth token to run the script.
> > > >
> > > > Please let me know if you have any other ideas to do this.
> > > >
> > > > Thanks,
> > > > Roshani
> > > >
> > >
> > >
> > >
> > > --
> > > Sandeep Krishnamurthy
> > >
> >
>


installation page UX

2018-05-16 Thread Aaron Markham
Hi,
I've written some notes on the wiki about issues with the installation page
along with some suggestions. I'd appreciate your feedback here or on the
wiki.

https://cwiki.apache.org/confluence/display/MXNET/Installation+page+UX

Cheers,
Aaron


Re: Pre-release versions on website

2018-05-07 Thread Aaron Markham
The release candidate was removed as an option on the site.

With regard to the default version, I posted about the predominance of
users on master, and all of the feedback I received agreed that master
should be the default view of the website.
If a change of heart occurs regarding the default view of the site, I made
the build_version_doc scripts, and Jenkins job that uses them, easily
configurable, so from the Jenkins job, one may simply change the "site
default" parameter on the nightly site build job. Viola, whatever version
you want as default will appear upon the next run. For now, however, I
think we should stick to what the users on the site seem to want, and
that's master.

That being said, the default installation instructions should reflect the
current release: pip install mxnet. They currently reflect how to install
master, using: --pre. I plan to remedy this soon. Anyone coming to the site
and wanting to get started should see something similar to how pytorch does
it. Their instructions are very clean and simple. Complex installation
information can be posted on a detail pages, but basic install info should
be like one line: pip install mxnet or pip install mxnet-{some-variant}.

Krishnan and I are working on a refactor of the install page to make it
easier to maintain and easier for visitors to the site to get started.
Also, the API docs should have the option to be easily switched and
searching should work within the version you have selected. Again, this is
something Krishnan and I are discussing and attempting to fix. I believe
that between these two updates, we'll have resolved the primary UX problems
with the versions selector, as well as allayed the concerns about clarity
around the API docs. You should know what version of content you're reading
at any point in time. I'm happy to hear any suggestions or requests on this
area as we're in the middle of mocking up some solutions.

Cheers,
Aaron


On Mon, May 7, 2018 at 8:03 AM, Hen <bay...@apache.org> wrote:

> Agreed.
>
> RCs are for the project contributors to review. They are not for users as
> they have not passed the release process. The website shouldn’t reflect
> RCs.
>
> Hen
>
> On Sat, May 5, 2018 at 10:42 PM Afrooze, Sina <sina@gmail.com> wrote:
>
> > I have mentioned this before and I still think that the default MXNet
> > webpage must match the default version that is installed when someone
> > issues "pip install mxnet". - Sina
> >
> >
> >
> > On 5/1/18, 6:43 PM, "Marco de Abreu" <marco.g.ab...@googlemail.com>
> > wrote:
> >
> > Hi Aaron,
> > I think it'd be a good idea to create pip packages for the release
> > candidates. Sheng, would it be possible to incorporate this into your
> > build
> > pipeline? In my opinion, we shouldn't advertise the RCs not too much
> > on our
> > website, so I'd say we could skip #2. #3 in combination with #1
> sounds
> > like
> > a good idea.
> >
> > In the meantime, we could just patch the install page for 1.2.0 and a
> > big
> > red warning message that states that this a release candidate and
> that
> > the
> > publish process is ongoing. This way, we can let the 1.2.0 website
> > stay up
> > and also avoid confusion. WDYT?
> >
> > -Marco
> >
> > On Wed, May 2, 2018 at 1:22 AM, Aaron Markham <
> > aaron.s.mark...@gmail.com>
> > wrote:
> >
> > > Hi Marco,
> > > Putting 1.2.0 was meant to allow a preview and for testing, but I
> > see how
> > > this can cause confusion.
> > >
> > > Change the names and/or hiding the RC is possible with some
> > refactoring of
> > > the site build scripts. I'd be happy to take a look at this.
> > >
> > > I think another solution could be:
> > > 1. create pip installs for each release candidate
> > > 2. offer each release candidate on the install page (with
> > instructions for
> > > source and pip installs)
> > > 3. when there's a dropdown for version, in either the install or
> API
> > docs
> > > area it maps precisely to the release or release candidate
> > >
> > > In the meantime, to prevent confusion, I'll remove 1.2.0 from the
> > site.
> > > Once we cut the release, or solved the points above, whichever is
> > sooner,
> > > the version(s) can be reintroduced to the site.
> > >
> > > Cheers,
> > > Aaron
> > >
> > > On Tue, May 1, 2018 at 3:26 PM, Marco de Abreu <
> > > marco.g.ab...@googlemail.com
> > > > wrote:
> > >
> > > > Hello,
> > > >
> > > > we have been approached by a few users about the release status
> of
> > MXNet
> > > > 1.2. One thing that seems to be confusing in particular is the
> > fact that
> > > we
> > > > already show version 1.2.0 on our website. Would it be possible
> to
> > > > blacklist (and remove) pre-release versions from website until
> > they have
> > > > been approved and published?
> > > >
> > > > Best regards,
> > > > Marco
> > > >
> > >
> >
> >
> >
> >
>


Re: Pre-release versions on website

2018-05-01 Thread Aaron Markham
Hi Marco,
Putting 1.2.0 was meant to allow a preview and for testing, but I see how
this can cause confusion.

Change the names and/or hiding the RC is possible with some refactoring of
the site build scripts. I'd be happy to take a look at this.

I think another solution could be:
1. create pip installs for each release candidate
2. offer each release candidate on the install page (with instructions for
source and pip installs)
3. when there's a dropdown for version, in either the install or API docs
area it maps precisely to the release or release candidate

In the meantime, to prevent confusion, I'll remove 1.2.0 from the site.
Once we cut the release, or solved the points above, whichever is sooner,
the version(s) can be reintroduced to the site.

Cheers,
Aaron

On Tue, May 1, 2018 at 3:26 PM, Marco de Abreu  wrote:

> Hello,
>
> we have been approached by a few users about the release status of MXNet
> 1.2. One thing that seems to be confusing in particular is the fact that we
> already show version 1.2.0 on our website. Would it be possible to
> blacklist (and remove) pre-release versions from website until they have
> been approved and published?
>
> Best regards,
> Marco
>


Re: mxnet.io website publishing

2018-04-25 Thread Aaron Markham
The MXNet website has been updated. Please let me know if there are any
issues.
It always seems to get hung up on big commits, so you have to publish a
tiny update. I've added this to the publish CI job via the date.txt file
which happens as a follow up commit to nudge the site replication process.
(Thanks for the idea, Marco).

Cheers,
Aaron

On Wed, Apr 25, 2018 at 2:16 PM, Aaron Markham <aaron.s.mark...@gmail.com>
wrote:

> I've updated Jenkins CI and the docs build scripts with #10485
> <https://github.com/apache/incubator-mxnet/pull/10485>. The CI run to
> build the website with the latest docs has completed successfully
> <http://jenkins.mxnet-ci.amazon-ml.com/job/website%20build%20-%20test%20pipeline/62/>,
> and it deployed the docs to my fork
> <https://github.com/aaronmarkham/incubator-mxnet-site/>. A preview of the
> site is hosted here: http://34.235.144.251/
>
> To get the site by refreshed by deploying to the production repo, I need
> credentials for the site repo at https://github.com/apache/
> incubator-mxnet-site. Then I'll rerun the publish job after pointing it
> at production.
>
> Who can grant me access to the incubator-mxnet-site repo? There are
> several accounts in CI's security tokens list that might work, but I don't
> want to use any of those without permission (and I'm not sure if they have
> access anyway). If you're setup and want to let me use yours, that's
> fine... just let me know.
>
> Cheers,
> Aaron
>
>
>
>
>


mxnet.io website publishing

2018-04-25 Thread Aaron Markham
I've updated Jenkins CI and the docs build scripts with #10485
. The CI run to build
the website with the latest docs has completed successfully
,
and it deployed the docs to my fork
. A preview of the
site is hosted here: http://34.235.144.251/

To get the site by refreshed by deploying to the production repo, I need
credentials for the site repo at
https://github.com/apache/incubator-mxnet-site. Then I'll rerun the publish
job after pointing it at production.

Who can grant me access to the incubator-mxnet-site repo? There are several
accounts in CI's security tokens list that might work, but I don't want to
use any of those without permission (and I'm not sure if they have access
anyway). If you're setup and want to let me use yours, that's fine... just
let me know.

Cheers,
Aaron


data.mxnet.io is gone (correction)

2018-04-13 Thread Aaron Markham
So anyone know what happened? It's a critical resource for datasets
and examples and more... and all the files seem gone.

http://data.mxnet.io/
That's show forbidden... which is unusual.

Then any resource like
http://data.mxnet.io/models/imagenet/squeezenet/ is missing.


mxnet.data.io is gone!?

2018-04-13 Thread Aaron Markham
So anyone know what happened? It's a critical resource for datasets
and examples and more... and all the files seem gone.

http://data.mxnet.io/
That's show forbidden... which is unusual.

Then any resource like
http://data.mxnet.io/models/imagenet/squeezenet/ is missing.


Re: blog for MXNet

2018-04-11 Thread Aaron Markham
I think for China we need to cross-post to WeChat. Apparently, there
is already blog post activity for MXNet there, and it would make sense
to be on that platform directly.

Can a China user access the Apache blog site that Hen mentioned? Also,
I'm not familiar with the Apache blog and how you contribute. I don't
see info about it on Confluence or elsewhere. Certainly sounds like
something that needs some attention and to be part of regular
communications.


On Wed, Apr 11, 2018 at 6:21 PM, Zhao, Patric <patric.z...@intel.com> wrote:
> FYI, China user can't access medium.com :(
>
>> -Original Message-
>> From: Anirudh Acharya [mailto:anirudhk...@gmail.com]
>> Sent: Thursday, April 12, 2018 6:31 AM
>> To: dev@mxnet.incubator.apache.org
>> Subject: Re: blog for MXNet
>>
>> There is already an AWS Evangelist, Julien Simon, who has quite a few posts
>> about mxnet/gluon on medium - https://medium.com/@julsimon
>>
>>
>> Regards
>> Anirudh
>>
>> On Wed, Apr 11, 2018 at 3:27 PM, Sebastian Gutierrez <
>> sebast...@aiworkbox.com> wrote:
>>
>> > Aaron and Thomas
>> >
>> > Great ideas!
>> >
>> > One thing worth also considering is something like
>> >
>> > https://www.r-bloggers.com/
>> >
>> > What it does is serve as a blog aggregation service for all of the
>> > people who have blogged about r topics. Because of the central
>> > repository nature, it serves as a natural gathering point and allows
>> > people not using RSS (or similar technologies) to keep up to date with 
>> > what is
>> happening.
>> >
>> > Another thing worth considering is a job board / site for MXNet full
>> > time / part time / remote jobs.  The data vis community has this free
>> > email list service
>> > https://groups.google.com/forum/m/#!forum/data-vis-jobs that's very
>> > community friendly and is a good place for people to gather to see job
>> > needs.
>> >
>> > All the best
>> > Sebastian Gutierrez
>> >
>> >
>> > On Wed, Apr 11, 2018 at 6:10 PM Thomas DELTEIL
>> > <thomas.delte...@gmail.com>
>> > wrote:
>> >
>> > > Thanks Aaron, I like medium, a lot of projects seems to be posting
>> > > their articles there, as you mentioned.
>> > >
>> > > Note that there is a newly created Chinese MXNet blog here:
>> > > https://zh.mxnet.io/blog/
>> > >
>> > > I would be happy to contribute to the blogs, if you want to add me
>> > > to the writer/editor list.
>> > >
>> > >
>> > >
>> > > Also, there is a mxnet subreddit r/mxnet which was created by
>> > > Sebastian
>> > > (thanks!) and I am now a moderator as well. Feel free to cross-post
>> > > any interesting content there! https://www.reddit.com/r/mxnet/
>> > > Please subscribe!
>> > >
>> > >
>> > > I will try to post one link a day at least, until I run out of links
>> > > ☺ We will also improve the look of it this week and add links to
>> > > relevant resources on the side bar, etc.
>> > >
>> > >
>> > >
>> > > All the best,
>> > >
>> > >
>> > > Thomas
>> > >
>> > >
>> > > 2018-04-11 14:45 GMT-07:00 Aaron Markham
>> <aaron.s.mark...@gmail.com>:
>> > >
>> > > > Having a blog for MXNet would be very useful for conveying news,
>> > > > talking about features, demoing applications, and building awareness.
>> > > >
>> > > > Does anyone have particular preferences or recommendations on blog
>> > > > hosting or platform?
>> > > >
>> > > > I currently have editor access for an MXNet branded account on Medium.
>> > > > https://medium.com/mxnet
>> > > >
>> > > > There's nothing there at the moment, but at least with Medium we
>> > > > all could get started right away, and have a built-in syndication
>> > > > platform. Also, note that this is where the TensorFlow blog resides:
>> > > > https://medium.com/tensorflow
>> > > >
>> > > > Please make it known if you'd like to contribute, so you can get
>> > > > writer/editor access (to whichever platform we settle on.)
>> > > >
>> > > > Cheers,
>> > > > Aaron
>> > > >
>> > >
>> >


blog for MXNet

2018-04-11 Thread Aaron Markham
Having a blog for MXNet would be very useful for conveying news,
talking about features, demoing applications, and building awareness.

Does anyone have particular preferences or recommendations on blog
hosting or platform?

I currently have editor access for an MXNet branded account on Medium.
https://medium.com/mxnet

There's nothing there at the moment, but at least with Medium we all
could get started right away, and have a built-in syndication
platform. Also, note that this is where the TensorFlow blog resides:
https://medium.com/tensorflow

Please make it known if you'd like to contribute, so you can get
writer/editor access (to whichever platform we settle on.)

Cheers,
Aaron


Re: MXNet Name Change?

2018-04-11 Thread Aaron Markham
Changing branding is hard as we already have some momentum under the
current name. It's not impossible, and if someone has a fantastic idea
and marketing plan for it, it's worth considering.

Aside from that, updating the pronunciation could be useful if you
like having those gif vs jif debates, but at least people would be
talking about it. I personally like the sound of "mix-net" over
"em-ex-net". Since we're just starting Youtube videos, I think it's a
great idea to establish the pronunciation right away in those videos.



On Wed, Apr 11, 2018 at 1:33 PM, Chris Olivier  wrote:
> Just curious why you think it’s a bad idea — you didn’t say?
>
> On Wed, Apr 11, 2018 at 12:49 PM Chen HY  wrote:
>
>> At least people needs a way to speak it.
>> Just define its pronunciation as "mix-net" or "m-x-net" and use the agreed
>> one everywhere helps a lot.
>> Changing name is a bad idea.
>>
>> 2018-04-11 20:29 GMT+01:00 Mu Li :
>>
>> > Agree that MXNet, the combination of Minerva and CXXNet, which can be
>> > interpreted as mixed-net, is hard to be pronounced. But rebranding a name
>> > is a very big decision. We need a very carefully designed marketing plan
>> > for it.
>> >
>> > A choice is that we can gradually refer MXNet as a backend, and talk more
>> > about the frontend Gluon.
>> >
>> > On Wed, Apr 11, 2018 at 12:16 PM, Thomas DELTEIL <
>> > thomas.delte...@gmail.com>
>> > wrote:
>> >
>> > > FWIW Brainscript is actually a network definition language:
>> > > https://docs.microsoft.com/en-us/cognitive-toolkit/
>> > > BrainScript-Network-Builder
>> > >
>> > >
>> > >
>> > > Thomas
>> > >
>> > >
>> > > 2018-04-11 12:13 GMT-07:00 Chiyuan Zhang :
>> > >
>> > > > IIRC CNTK renamed to something like brainscript which does not seem
>> to
>> > be
>> > > > very successful publicity campaign?
>> > > >
>> > > > Chiyuan
>> > > >
>> > > > On Wed, Apr 11, 2018 at 10:18 AM Chris Olivier <
>> cjolivie...@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > Should we consider renaming MXNet to something more "friendly"?
>> > > > >
>> > > > > IMHO, I think this may be related to adoption problems.
>> > > > >
>> > > > > MXNet, CMTK -- both seem sort of sterile and hard to use, don't
>> they?
>> > > > >
>> > > > > Tensorflow, PyTorch, Caffe -- sound cool.
>> > > > >
>> > > > --
>> > > > Semt ftom m ipohne
>> > > >
>> > >
>> >
>>
>>
>>
>> --
>> Chen Hanyang 陈涵洋
>> Software School Fudan University
>> +86-138-1881-7745
>>


Re: Display the master branch website in default for a period

2018-03-20 Thread Aaron Markham
Embedding images doesn't work, I guess. Here's a link to the chart.
https://drive.google.com/file/d/1GzxDEhF6tx9_bsFc4LuUzgoIaMSiEV5s/view?usp=sharing


On Tue, Mar 20, 2018 at 9:24 AM, Aaron Markham <aaron.s.mark...@gmail.com>
wrote:

> I would like to get a sense of people's feelings with regard to actual
> data on the usage of the site.
> *Users prefer master*
> Despite it defaulting to version 1.1.0, nearly 60% of the page views are
> on master. I think it is pretty clear what the website users want.
> I've implemented a new site building script
> <https://github.com/apache/incubator-mxnet/tree/master/docs/build_version_doc#build_site_tagsh>
> that can easily swap the default version to any tag, and hope to get that
> integrated with the CI process this week. We could go live with master
> being default whenever it is agreed.
>
> *Time-travel website is difficult to maintain, confuses search, and is a
> strange user experience*
> Switching versions on the whole site and maintaining these time-traveling
> website builds is not really worth the effort and these old sites
> introduces a lot of problems with search and poor information. (+1 to
> Christopher's sentiments on UX).
> I think that we should keep the versions of MXNet limited to the install
> page(s) and the API docs. This is where versions have value and user
> traffic. This accounts to over 50% of all traffic in the legacy versions,
> and 80%+ of this traffic is coming directly from search, primarily Google.
> We can enhance visibility to these data points by enabling the Google
> Analytics Search Console, but in absence of that, experience suggests that
> if old content is available and linked to, it reinforces itself despite it
> being deprecated or even inaccurate. By stashing versioned content where it
> is accessible, but not highlighted, we'll force the search engines to
> update their links, and over time, any dent to the ranking of search
> results will heal, and the site's primary and current content will appear
> at the top of the results.
>
> *Maintaining tutorials and examples in master is easier, will help search,
> and provide a better user experience*
> There's another 15-25% of legacy version traffic going to tutorials, and
> all of it comes from search. These old tutorials are not maintained and
> while they might theoretically work with the specific API version they're
> coupled with in the build, they are also riddled with broken links, missing
> datasets and Python 3 incompatibilities. IMO, we should flag each tutorial
> in master with the minimum required API version and Python version, and no
> longer support legacy tutorials as a matter of course. If someone wants to
> fix them, then great. They can make the tutorial in master backwards
> compatible, or create a separate tutorial that focuses on the legacy
> version. But it is maintained in master. This shift will force search to
> update, guiding users to working tutorials and fresh content.
>
> In conclusion there are three overlapping proposals here.
> 1. Make master primary.
> 2. Remove time travel from the website. Provide specific instructions on
> installing master, current release, and a subset of legacy versions.
> Provide versioned API docs on the website. People can still download tagged
> releases and build the old site and docs if they wish.
> 3. Maintain tutorials and examples in master.
>
> Cheers,
> Aaron
>
> P.S. Any moves or removal of content will be handled by 301 permanent
> redirects, so we can soften the transition.
>
>
> On Mar 1, 2018 19:55, "Barber, Christopher" <christopher.bar...@analog.com>
> wrote:
>
>> I was thinking more along the lines of benchmarks of MXNet vs TensorFlow,
>> PyTorch, and Caffe2. Benchmarks of edge devices would definitely be
>> interesting, but I would also want to see benchmarks of training time and
>> memory use and accuracy on large models. Obviously this would be a
>> non-trivial amount of work, which is why no one else is doing it, but there
>> would be a lot of interest in this. Also would like to see benchmarks of
>> ndarray, vs symbol vs gluon.
>>
>> But yes, if you want to drive traffic to the website you should have
>> content that changes frequently. I have to say I find it really strange to
>> have the entire website change when I select a different version from the
>> top tab. The design of the website should be independent of the code
>> version.
>>
>> - Christopher
>>
>> On 3/1/18, 4:33 PM, "Marco de Abreu" <marco.g.ab...@googlemail.com>
>> wrote:
>>
>> As far as I know, there are plans to make regular benchmarks and
>> generate
>> stat

Re: Display the master branch website in default for a period

2018-03-20 Thread Aaron Markham
ebsite contain
> static
> parts (e.g. APIs and documentation) as well as dynamic parts (e.g.
> news,
> statistics, recent papers etc).
>
> How does that sound?
>
> -Marco
>
> Aaron Markham <aaron.s.mark...@gmail.com> schrieb am Do., 1. März
> 2018,
> 22:11:
>
> > Do you have specific public benchmarks in mind?
> >
> > On Mar 1, 2018 10:13, "Barber, Christopher" <
> christopher.bar...@analog.com
> > >
> > wrote:
> >
> > > I think one thing that could draw more users would be published
> > benchmarks
> > > that show that networks implemented using MXNet perform better than
> > > comparable ones using other platforms using the same hardware. If
> you can
> > > definitively show that MXNet is much faster and/or uses much less
> memory,
> > > you will attract much more interest.
> > >
> > > - Christopher
> > >
> > > On 2/25/18, 11:53 PM, "Li, Mu" <m...@amazon.com> wrote:
> > >
> > > That's a great idea. The thing Simon's team is doing is
> publishing
> > > more tutorials on both MXNet website and AWS blog, which may
> attract a
> > lot
> > > of traffics. Also Sukwon is tracking the progress of publishing
> news more
> > > frequently on the homepage.
> > >
> > > Best,
> > > Mu
> > >
> > > > On Feb 25, 2018, at 8:48 PM, Chris Olivier <
> cjolivie...@gmail.com>
> > > wrote:
> > > >
> > > > My (probably less-than-$0.02):
> > > >
> > > > I have as my home page on my
> > > > phone, the Google Research Blog, and they frequently release
> stuff
> > > like
> > > > data sets and models do this or that.  Usually it seems
> pretty
> > > interesting
> > > > and I am compelled to try it.
> > > >
> > > > Maybe we do something similar, but I’m not aware of it. I
> know we
> > > have all
> > > > sorts of examples and whatnot, but it doesn’t seem the same
> as what
> > > at
> > > > least appears to be something new to play with scrolling
> past every
> > > couple
> > > > of weeks:
> > > >
> > > > For example, a few days ago:
> > > >
> > > > “Introducing the HDR+ Burst Photography Dataset”.
> > > >
> > > > https://research.googleblog.com/2018/02/introducing-hdr-
> > > burst-photography.html?m=1
> > > >
> > > > Reading that makes me want to download it and play around.
> > > Obviously I
> > > > would use Tensorflow by default because it’s ready to roll
> as-is
> > > with this
> > > > dataset/model.
> > > >
> > > >
> > > > -Chris
> > > >
> > > >
> > > >
> > > >> On Sun, Feb 25, 2018 at 8:34 PM Mu Li <limu...@gmail.com>
> wrote:
> > > >>
> > > >> Not sure why the screenshot of the page view is not there,
> attach
> > > it again:
> > > >>
> > > >>
> > > >> Best,
> > > >> Mu
> > > >>
> > > >>> On Feb 25, 2018, at 8:32 PM, Li, Mu <m...@amazon.com>
> wrote:
> > > >>>
> > > >>> Hi Team,
> > > >>>
> > > >>> We are in trouble attracting new users. According to Google
> > > analytics,
> > > >> there is almost no increase in the number of paper views
> for the
> > > document
> > > >> site mxnet.io.
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>> The number of paper views is an important metric for the
> > adoption,
> > > I
> > > >> would like to take actions to improve this number. It
> includes
> > > improving
> > > >> the website so that users can get information easier.
> However, the
> > > current
> > > >> website displays the last stable version instead of the
> master
> > > branch. Then
> > > >> the effect of a modification, namely the user behaviors,
> may need
> > a
> > > few
> > > >> months to observe, which is definitely not effective.
> > > >>>
> > > >>> One aggressive idea is just showing the master branch in
> default
> > > during
> > > >> the website improvement period (may take 3 months). Another
> way is
> > > >> releasing more frequently, e.g a new release per 2 weeks.
> > > >>>
> > > >>> What's your thoughts?
> > > >>>
> > > >>> Best,
> > > >>> Mu
> > > >>
> > >
> > >
> > >
> > >
> >
>
>
>


Re: [RESULT][VOTE] tracking code changes with JIRA by associating pull requests

2018-03-07 Thread Aaron Markham
I'm not quite sure if I have enough background on reasons for or against
this to vote in the next round, but my two cents: I didn't see much debate
on why we need yet another tool for issues that we have to manually
maintain...the vote kind of slid in there without many stakeholders
realizing what they were being signed up for. I was thinking, sure, if YOU
want to make jira tickets, go right ahead. I have two internal ticketing
systems to deal with already that assign tasks on MXNet, plus GitHub. Jira
would be four. Happy to make it work, but I'll need fifth tool to aggregate
communications and metrics between the other four tools! I'm only sort of
joking.

I saw Chris's response, and ok the public visibility part makes sense, but
does this phase out any other overhead? Does it integrate? Jira has
integration options so maybe we can eliminate some overhead... Like
something that hooks into the GitHub api and generates jira tickets on the
fly... I want to believe there's a plan that makes this all easier.

What value I don't see is how we lower barriers to contribution and make it
more fluid for new users that could become contributors. What's the story
and value proposition?

Also, I don't see any docs on the website or on github on how to sign up
for jira, or how to contribute according to this new requirement anywhere
on the site. Myself and new contributors wouldn't know what the expected
flow looks like because it's not really accessible. I now see the
confluence wiki, but that's pretty much hidden from anyone browsing the
site or github and looking to contribute. Why is this info on confluence at
all? Why not in the docs on github that are rendered to the website? Or
conversely, why is some of the info on github and on the website, if it is
being maintained and current only on confluence?

These are two separate issues really, but I think if you want buy-in, this
needs to be more transparent and obvious, and provide clear reasons and
benefits to why you're asking for more overhead.

Aaron

On Mar 6, 2018 21:14, "Eric Xie"  wrote:

> -1
>
> JIRA is ancient and arcane. This adds unnecessary overhead.
>
> On 2018/03/03 06:11:12, CodingCat  wrote:
> > This vote passes with 6 +1 votes (6 bindings) and no 0 or -1 votes.
> >
> > Binding +1:
> > Chris Olivier
> > Indhu Bharathi
> > Suneel Marthi
> > Yuan Tang
> > Marco de Abreu
> > Sebastian Schelter
> >
> >
> >
> > Vote thread:
> > https://lists.apache.org/list.html?d...@mxnet.apache.org:lte=
> 1M:tracking%20code%20changes%20with%20JIRA%20by%20associatin
> g%20pull%20requests
> >
> > I will continue with pushing the content to wiki and take it into
> practice
> >
>


404 issues

2018-03-05 Thread Aaron Markham
I've been notified by several parties about 404s for files that are
now longer available on the site.
https://github.com/apache/incubator-mxnet/issues/9917

This page returns 404:
https://mxnet.incubator.apache.org/api/python/module.html

It was moved here:
https://mxnet.incubator.apache.org/api/python/module/module.html

There are many other examples of moved content. Some are temporarily
"fixed" via adding a meta-refresh tag in the html source for the old
pages. For example the meta tag is being used to redirect to faq, the
new location for the how_to docs.

It would seem a better solution for us is to use htaccess files and
publish persistent redirects for the new location(s) of content.

Do we have a way of pushing a config to the Apache infra to facilitate
this? I think we'd need config access if we're to put up some custom
404 pages too (which would be nicer than what we have now.)

Also, is there a good way to access the log files to get a better idea
of the 404 situation?

Cheers,
Aaron


Re: Display the master branch website in default for a period

2018-03-01 Thread Aaron Markham
Do you have specific public benchmarks in mind?

On Mar 1, 2018 10:13, "Barber, Christopher" 
wrote:

> I think one thing that could draw more users would be published benchmarks
> that show that networks implemented using MXNet perform better than
> comparable ones using other platforms using the same hardware. If you can
> definitively show that MXNet is much faster and/or uses much less memory,
> you will attract much more interest.
>
> - Christopher
>
> On 2/25/18, 11:53 PM, "Li, Mu"  wrote:
>
> That's a great idea. The thing Simon's team is doing is publishing
> more tutorials on both MXNet website and AWS blog, which may attract a lot
> of traffics. Also Sukwon is tracking the progress of publishing news more
> frequently on the homepage.
>
> Best,
> Mu
>
> > On Feb 25, 2018, at 8:48 PM, Chris Olivier 
> wrote:
> >
> > My (probably less-than-$0.02):
> >
> > I have as my home page on my
> > phone, the Google Research Blog, and they frequently release stuff
> like
> > data sets and models do this or that.  Usually it seems pretty
> interesting
> > and I am compelled to try it.
> >
> > Maybe we do something similar, but I’m not aware of it. I know we
> have all
> > sorts of examples and whatnot, but it doesn’t seem the same as what
> at
> > least appears to be something new to play with scrolling past every
> couple
> > of weeks:
> >
> > For example, a few days ago:
> >
> > “Introducing the HDR+ Burst Photography Dataset”.
> >
> > https://research.googleblog.com/2018/02/introducing-hdr-
> burst-photography.html?m=1
> >
> > Reading that makes me want to download it and play around.
> Obviously I
> > would use Tensorflow by default because it’s ready to roll as-is
> with this
> > dataset/model.
> >
> >
> > -Chris
> >
> >
> >
> >> On Sun, Feb 25, 2018 at 8:34 PM Mu Li  wrote:
> >>
> >> Not sure why the screenshot of the page view is not there, attach
> it again:
> >>
> >>
> >> Best,
> >> Mu
> >>
> >>> On Feb 25, 2018, at 8:32 PM, Li, Mu  wrote:
> >>>
> >>> Hi Team,
> >>>
> >>> We are in trouble attracting new users. According to Google
> analytics,
> >> there is almost no increase in the number of paper views for the
> document
> >> site mxnet.io.
> >>>
> >>>
> >>>
> >>>
> >>> The number of paper views is an important metric for the adoption,
> I
> >> would like to take actions to improve this number. It includes
> improving
> >> the website so that users can get information easier. However, the
> current
> >> website displays the last stable version instead of the master
> branch. Then
> >> the effect of a modification, namely the user behaviors, may need a
> few
> >> months to observe, which is definitely not effective.
> >>>
> >>> One aggressive idea is just showing the master branch in default
> during
> >> the website improvement period (may take 3 months). Another way is
> >> releasing more frequently, e.g a new release per 2 weeks.
> >>>
> >>> What's your thoughts?
> >>>
> >>> Best,
> >>> Mu
> >>
>
>
>
>