Join slack channel

[leonardo espinosa]

Hi there,

I'm using MXNet for teaching and research, I'd love to join
the slack channel for some questions.

Thanks,

---
Dr. Leonardo Espinosa
aboutMe: 
http://www.espinosaleal.me

Re: Join slack channel

[TongKe Xue]

Hi,

  I would also like to join the slack channel. I'm trying to use mxnet
in production. (Java API seems most production ready among all DL
libraries.)

--TongKe

On Tue, Nov 7, 2017 at 1:11 AM, leonardo espinosa
 wrote:
> Hi there,
>
> I'm using MXNet for teaching and research, I'd love to join
> the slack channel for some questions.
>
> Thanks,
>
> ---
> Dr. Leonardo Espinosa
> aboutMe: 
> http://www.espinosaleal.me

Data Pipeline Intermediate Representation in MXNet/NNVM

[Yao Wang]

Hi,

Tensorflow has a transform package
https://github.com/tensorflow/transform which
is capable of export a data preprocessing pipeline to a tensorflow graph,
which can be incorporated into network graph. This package provides a neat
way to manage data pipeline together with network graph, since these data
process graph can be easily reused by other developers. Also I think we can
get some performance improvement by using computation graph for data
process rather than imperative processing for large data stream?

Currently in MXNet, if I want to do the similar thing, I need to pack the
code(most time python script) directly with network graph files. This
method has some issues:
1. Potential security issue. If I wrote the processing codes and I am the
only person use it, it's fine. However, if someone else wants to reuse it
in their application, they need to check the code to make sure there is no
security issue. It is not quite portable for reusing.

2. It is bind to specific language. Usually it's easier to develop deep
learning application using python, but if my production environment doesn't
have python environment, I need to either setup python environment or
rewrite this script with the language supported by my production
environment.

Any thought about supporting data pipeline IR in MXNet/NNVM?

[VOTE] Release Apache MXNet(incubating) version 0.12.1.rc0

[Meghna Baijal]

This is the vote to release Apache MXNet (incubating) version 0.12.1.

Voting will start now (Tuesday, November 7, 2017) and

close Friday, November 10, 2017


Link to release notes:

https://cwiki.apache.org/confluence/display/MXNET/MXNet+0.12.1+Release+Notes


Link to release candidate 0.12.1.rc0:

https://dist.apache.org/repos/dist/dev/incubator/mxnet/0.12.1.rc0/


View this page and scroll down to “Build from Source” to build this project:

https://mxnet.incubator.apache.org/install/index.html


The release tag can be found here:

https://github.com/apache/incubator-mxnet/tree/0.12.1.rc0

(Note: The README.md points to the 0.12.1 tag and does not work at the
moment.)



Please make sure you TEST before you vote accordingly:


+1 = approve


+0 = no opinion


-1 = disapprove (provide reason)



Thanks,

Meghna Baijal

Re: [VOTE] Release Apache MXNet(incubating) version 0.12.1.rc0

[Chris Olivier]

+1 Binding


On Tue, Nov 7, 2017 at 7:48 PM Meghna Baijal 
wrote:

> This is the vote to release Apache MXNet (incubating) version 0.12.1.
>
> Voting will start now (Tuesday, November 7, 2017) and
>
> close Friday, November 10, 2017
>
>
> Link to release notes:
>
>
> https://cwiki.apache.org/confluence/display/MXNET/MXNet+0.12.1+Release+Notes
>
>
> Link to release candidate 0.12.1.rc0:
>
> https://dist.apache.org/repos/dist/dev/incubator/mxnet/0.12.1.rc0/
>
>
> View this page and scroll down to “Build from Source” to build this
> project:
>
> https://mxnet.incubator.apache.org/install/index.html
>
>
> The release tag can be found here:
>
> https://github.com/apache/incubator-mxnet/tree/0.12.1.rc0
>
> (Note: The README.md points to the 0.12.1 tag and does not work at the
> moment.)
>
>
>
> Please make sure you TEST before you vote accordingly:
>
>
> +1 = approve
>
>
> +0 = no opinion
>
>
> -1 = disapprove (provide reason)
>
>
>
> Thanks,
>
> Meghna Baijal
>

[BUILD FAILED] Branch master build 603

[Apache Jenkins Server]

Build for MXNet branch master has broken. Please view the build at 
https://builds.apache.org/job/incubator-mxnet/job/master/603/

[BUILD FAILED] Branch master build 602

[Apache Jenkins Server]

Build for MXNet branch master has broken. Please view the build at 
https://builds.apache.org/job/incubator-mxnet/job/master/602/

Re: [VOTE] Release Apache MXNet(incubating) version 0.12.1.rc0

[Sebastian]

+1 (binding)

verified signature, built from source.

On 08.11.2017 04:51, Chris Olivier wrote:

+1 Binding


On Tue, Nov 7, 2017 at 7:48 PM Meghna Baijal 
wrote:


This is the vote to release Apache MXNet (incubating) version 0.12.1.

Voting will start now (Tuesday, November 7, 2017) and

close Friday, November 10, 2017


Link to release notes:


https://cwiki.apache.org/confluence/display/MXNET/MXNet+0.12.1+Release+Notes


Link to release candidate 0.12.1.rc0:

https://dist.apache.org/repos/dist/dev/incubator/mxnet/0.12.1.rc0/


View this page and scroll down to “Build from Source” to build this
project:

https://mxnet.incubator.apache.org/install/index.html


The release tag can be found here:

https://github.com/apache/incubator-mxnet/tree/0.12.1.rc0

(Note: The README.md points to the 0.12.1 tag and does not work at the
moment.)



Please make sure you TEST before you vote accordingly:


+1 = approve


+0 = no opinion


-1 = disapprove (provide reason)



Thanks,

Meghna Baijal

The process for a patch release

[Meghna Baijal]

Hi All,
A patch release is being planned for Apache MXNet(incubating) to include
some important bug fixes.
I want to confirm that for a patch release, there is no RC and voting step.

Effectively the process looks something like this -

Step 1: Cherrypick all the necessary bugfixes into the existing release
branch.

Step 2: Add a PR to update the version number etc (Example
https://github.com/apache/incubator-mxnet/commit/e0c7906693f0c79b0ce34a4d777c26a6bf1903c1
)

Step 3: Make sure the release branch build passes on builds.apache.jenkins
and nightly tests

Step 4: tag the release branch in GitHub with the release tag - example
0.12.1

Step 5: Download from the repo and checkout the tag, package it and Sign
the src code into 'apache-mxnet-src-0.12.1-incubating'

Step 6: Validate the signatures and test

Step 7: Update the website. (pip and docker release)

Step 8: Upload the src tar to the final destination -
http://www.apache.org/dist/incubator/mxnet/

Step 9: Announce the patch release

Thanks,
Meghna Baijal

[BUILD FAILED] Branch master build 597

[Apache Jenkins Server]

Build for MXNet branch master has broken. Please view the build at 
https://builds.apache.org/job/incubator-mxnet/job/master/597/

Re: MXNet Slack channel

[de Abreu, Marco]

Added

On 07.11.17, 12:58, "Bay, Daniel"  wrote:

Requesting an invitation to the MXNet slack channel.

Thanks,
Daniel

[BUILD FAILED] Branch master build 598

[Apache Jenkins Server]

Build for MXNet branch master has broken. Please view the build at 
https://builds.apache.org/job/incubator-mxnet/job/master/598/

Re: The process for a patch release

[Chris Olivier]

Do we need a vote for a critical patch release?

On Tue, Nov 7, 2017 at 12:08 PM, Meghna Baijal 
wrote:

> Hi All,
> A patch release is being planned for Apache MXNet(incubating) to include
> some important bug fixes.
> I want to confirm that for a patch release, there is no RC and voting step.
>
> Effectively the process looks something like this -
>
> Step 1: Cherrypick all the necessary bugfixes into the existing release
> branch.
>
> Step 2: Add a PR to update the version number etc (Example
> https://github.com/apache/incubator-mxnet/commit/
> e0c7906693f0c79b0ce34a4d777c26a6bf1903c1
> )
>
> Step 3: Make sure the release branch build passes on builds.apache.jenkins
> and nightly tests
>
> Step 4: tag the release branch in GitHub with the release tag - example
> 0.12.1
>
> Step 5: Download from the repo and checkout the tag, package it and Sign
> the src code into 'apache-mxnet-src-0.12.1-incubating'
>
> Step 6: Validate the signatures and test
>
> Step 7: Update the website. (pip and docker release)
>
> Step 8: Upload the src tar to the final destination -
> http://www.apache.org/dist/incubator/mxnet/
>
> Step 9: Announce the patch release
>
> Thanks,
> Meghna Baijal
>

Data Pipeline Intermediate Representation in MXNet/NNVM

[Yao Wang]

Hi,

Tensorflow has a transform package https://github.com/tensorflow/transform
which is capable of export a data preprocessing pipeline to a tensorflow
graph, which can be incorporated into network graph. This package provides
a neat way to manage data pipeline together with network graph, since these
data process graph can be easily reused by other developers. Also I think
we can get some performance improvement by using computation graph for data
process rather than imperative processing for large data stream?

Currently in MXNet, if I want to do the similar thing, I need to pack the
code(most time python script) directly with network graph files. This
method has some issues:
1. Potential security issue. If I wrote the processing codes and I am the
only person use it, it's fine. However, if someone else wants to reuse it
in their application, they need to check the code to make sure there is no
security issue. It is not quite portable for reusing.

2. It is bind to specific language. Usually it's easier to develop deep
learning application using python, but if my production environment doesn't
have python environment, I need to either setup python environment or
rewrite this script with the language supported by my production
environment.

Any thought about supporting data pipeline IR in MXNet/NNVM?

Re: The process for a patch release

[Chris Olivier]

I don’t recall where I heard that. Oh well, no worries. Vote it is.

On Tue, Nov 7, 2017 at 5:28 PM Hen  wrote:

> I thought a patch release was exactly the same as a full release; except
> that there is less discussion of the features involved.
>
> Did you see a more lightweight release process described somewhere? I'd
> expect an RC and vote.
>
> Hen
>
> On Tue, Nov 7, 2017 at 12:08 PM, Meghna Baijal  >
> wrote:
>
> > Hi All,
> > A patch release is being planned for Apache MXNet(incubating) to include
> > some important bug fixes.
> > I want to confirm that for a patch release, there is no RC and voting
> step.
> >
> > Effectively the process looks something like this -
> >
> > Step 1: Cherrypick all the necessary bugfixes into the existing release
> > branch.
> >
> > Step 2: Add a PR to update the version number etc (Example
> > https://github.com/apache/incubator-mxnet/commit/
> > e0c7906693f0c79b0ce34a4d777c26a6bf1903c1
> > )
> >
> > Step 3: Make sure the release branch build passes on
> builds.apache.jenkins
> > and nightly tests
> >
> > Step 4: tag the release branch in GitHub with the release tag - example
> > 0.12.1
> >
> > Step 5: Download from the repo and checkout the tag, package it and Sign
> > the src code into 'apache-mxnet-src-0.12.1-incubating'
> >
> > Step 6: Validate the signatures and test
> >
> > Step 7: Update the website. (pip and docker release)
> >
> > Step 8: Upload the src tar to the final destination -
> > http://www.apache.org/dist/incubator/mxnet/
> >
> > Step 9: Announce the patch release
> >
> > Thanks,
> > Meghna Baijal
> >
>

Re: update build instructions

[Pedro Larroy]

Hi

I updated the build instructions using CMake, python3 and other minor
inacuracies in this PR:
https://github.com/apache/incubator-mxnet/pull/8578

Please have a look and comment.

I also added a file in the root "DEVELOPMENT.md"  which synthesizes
how to build mxnet and run the unit tests to get started making
changes.

Your feedback is welcome. And if you can please test the instructions
to check they work properly and nothing was missed.

Pedro.

On Thu, Nov 2, 2017 at 5:07 PM, Pedro Larroy
 wrote:
> Right.
>
> I tried now with the flavor that you requested and I have problems
> generating the build file:
>
> Seems that I need the variable pslite_LINKER_LIBS_DEBUG which is not
> set. Any idea on how to compile with this flavour? (dist KVSTORE)
>
>
>
>
> ubuntu@ip-172-31-35-161:~/mxnet/build$ cmake -DUSE_CUDA=ON
> -DUSE_DIST_KVSTORE=ON -GNinja ..
> -- Found MKL (include: /usr/local/include, lib: /usr/local/lib/libmklml_gnu.so
> -- Found OpenBLAS libraries: /usr/local/lib/libopenblas.so
> -- Found OpenBLAS include: /usr/local/include
> -- CUDA detected: 8.0
> -- Found cuDNN (include: /usr/local/cuda/include, library:
> /usr/local/cuda/lib64/libcudnn.so)
> -- Added CUDA NVCC flags for: sm_52
> -- Could NOT find Gperftools (missing:  GPERFTOOLS_LIBRARIES
> GPERFTOOLS_INCLUDE_DIR)
> -- Could NOT find Jemalloc (missing:  JEMALLOC_LIBRARY JEMALLOC_INCLUDE_DIR)
> --  OpenCV_LIBS=opencv_core;opencv_highgui;opencv_imgproc;opencv_imgcodecs
> -- OpenCV found (/usr/local/share/OpenCV)
> -- Could NOT find Jemalloc (missing:  JEMALLOC_LIBRARY JEMALLOC_INCLUDE_DIR)
> -- Found cuDNN (include: /usr/local/cuda/include, library:
> /usr/local/cuda/lib64/libcudnn.so)
> You have called ADD_LIBRARY for library mxnet without any source
> files. This typically indicates a problem with your CMakeLists.txt
> file
> -- Found PROTOBUF Compiler: /usr/local/bin/protoc
> CMake Error at CMakeLists.txt:446 (target_link_libraries):
>   The "debug" argument must be followed by a library.
>
>
> -- Configuring incomplete, errors occurred!
> See also "/home/ubuntu/mxnet/build/CMakeFiles/CMakeOutput.log".
> See also "/home/ubuntu/mxnet/build/CMakeFiles/CMakeError.log".
>
>
> Regards.
>
> Pedro.
>
> On Thu, Nov 2, 2017 at 4:52 PM, Bhavin Thaker  wrote:
>> I agree about your point on correctness -- do you know of any known
>> correctness issues with Ninja?
>>
>> These build times seem to be NOT with GPU builds and distributed kvstore
>> enabled -- could you please confirm? nvcc builds take a significant time.
>>
>> Bhavin Thaker.
>>
>> On Thu, Nov 2, 2017 at 8:45 AM, Pedro Larroy 
>> wrote:
>>
>>> Hi
>>>
>>> For me it's more about correctness and reproducibility than build
>>> times, nonetheless, seems that the ninja build is significantly faster
>>> than the Make build:
>>>
>>> Make:
>>>
>>> real4m32.779s
>>> user43m33.236s
>>> sys 0m52.940s
>>>
>>> CMake + Ninja:
>>>
>>> real3m30.794s
>>> user36m2.564s
>>> sys 0m56.368s
>>>
>>> Compiled on an g3.4xlarge with ebs
>>>
>>> Provisioned IOPS SSD
>>>
>>> io (115000 iops)
>>>
>>>
>>> Pedro.
>>>
>>> On Thu, Nov 2, 2017 at 4:07 PM, Bhavin Thaker 
>>> wrote:
>>> > Hi Pedro,
>>> >
>>> > Using Ninja to improve build times is a good suggestion. Can you share
>>> the
>>> > build times you have observed with and without using Ninja? I presume you
>>> > have enabled compile-time options for GPU builds and Distributed MXNet
>>> for
>>> > the builds you have experimented with.
>>> >
>>> > See also:
>>> > https://ninja-build.org/manual.html
>>> >
>>> > Thanks,
>>> > Bhavin Thaker.
>>> >
>>> > On Thu, Nov 2, 2017 at 7:57 AM, Pedro Larroy <
>>> pedro.larroy.li...@gmail.com>
>>> > wrote:
>>> >
>>> >> Hi
>>> >>
>>> >> I would like to update the MXNet build instructions.
>>> >>
>>> >> In particular I was thinking that it would be a good idea to update
>>> >> the instructions to use CMake + Ninja. And add more information about
>>> >> the different build flavours.
>>> >>
>>> >>
>>> >> https://mxnet.incubator.apache.org/install/index.html
>>> >>
>>> >>
>>> >> Thoughts?
>>> >>
>>>

Re: Disable 'required status check' for master

[Eric Xie]

Its the failed ones that's the problem. They fail because CI is not working 
properly. 
For example this trivial change to doc cannot be merged because CI failed: 
https://github.com/apache/incubator-mxnet/pull/8555/files

On 2017-10-31 13:36, Chris Olivier  wrote: 
> I see 73 PR's for mxnet.
> 
> 17 of those have successfully passed CI and are waiting for merge (assuming
> the CR passed human inspection).
> 21 Show some form of CI in progress.
> *Only 8 are in the queue in Jenkins*
> 
> The rest have failed CI for one reason or another.
> 
> 
> 
> 
> On Tue, Oct 31, 2017 at 1:17 PM, Eric Xie  wrote:
> 
> > The number of pending PRs is growing very fast. At the current rate it
> > will reach 200 before we can fix jenkins.
> >
> > Stability is not the only goal. Master will be most stable if we don't
> > push anything, but that's not what we want.
> >
> > Committers should be responsible for their commits. Good judgement is the
> > ultimate guarantee of good code.
> >
> > Thanks,
> > Eric
> >
> > On 2017-10-31 12:38, Chris Olivier  wrote:
> > > -1
> > >
> > > I personally think it's a necessary evil and a good forcing factor to get
> > > CI fixed.
> > > Before that requirements, things that failed unit tests were being pushed
> > > into master daily.  It was a big problem.  At least now master is stable.
> > >
> > > On Tue, Oct 31, 2017 at 12:22 PM, Hen  wrote:
> > >
> > > > This makes sense to me. CI isn't of value if it isn't continuous. +1.
> > > >
> > > > On Tue, Oct 31, 2017 at 12:07 PM, Indhu 
> > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > We have been having issues getting CI to work fast enough. Currently
> > the
> > > > CI
> > > > > is being the bottleneck to get commits in. This is severely impacting
> > > > > development. I propose we disable 'required status check' for the
> > master
> > > > > branch so that we can development is not impacted. We can work on
> > fixing
> > > > > the CI situation in parallel.
> > > > >
> > > > > Committer,
> > > > > Please vote if you are okay with disabling 'required status check'
> > for
> > > > > master.
> > > > >
> > > > > Once we have enough votes, I'll request a mentor to file a ticket
> > with
> > > > > infra.
> > > > >
> > > > > Thanks,
> > > > > Indu
> > > > >
> > > >
> > >
> >
> 

Re: The process for a patch release

[Hen]

Yes. You would just do it quicker (if such was truly needed). You would
still need at least 3 +1s, and at the moment there is the crutch that you
would also need 3 Incubator PMC +1s (or the first 3 would have to be IPMC).

If it's a security vulnerability, then we would use private@ and confer
with security@ on the process.

Hen

On Tue, Nov 7, 2017 at 1:45 PM, Chris Olivier  wrote:

> Do we need a vote for a critical patch release?
>
> On Tue, Nov 7, 2017 at 12:08 PM, Meghna Baijal  >
> wrote:
>
> > Hi All,
> > A patch release is being planned for Apache MXNet(incubating) to include
> > some important bug fixes.
> > I want to confirm that for a patch release, there is no RC and voting
> step.
> >
> > Effectively the process looks something like this -
> >
> > Step 1: Cherrypick all the necessary bugfixes into the existing release
> > branch.
> >
> > Step 2: Add a PR to update the version number etc (Example
> > https://github.com/apache/incubator-mxnet/commit/
> > e0c7906693f0c79b0ce34a4d777c26a6bf1903c1
> > )
> >
> > Step 3: Make sure the release branch build passes on
> builds.apache.jenkins
> > and nightly tests
> >
> > Step 4: tag the release branch in GitHub with the release tag - example
> > 0.12.1
> >
> > Step 5: Download from the repo and checkout the tag, package it and Sign
> > the src code into 'apache-mxnet-src-0.12.1-incubating'
> >
> > Step 6: Validate the signatures and test
> >
> > Step 7: Update the website. (pip and docker release)
> >
> > Step 8: Upload the src tar to the final destination -
> > http://www.apache.org/dist/incubator/mxnet/
> >
> > Step 9: Announce the patch release
> >
> > Thanks,
> > Meghna Baijal
> >
>

Re: The process for a patch release

[Chris Olivier]

Thank you for the clarification!  The security release process certainly
makes sense (not to publicly announce security holes).

On Tue, Nov 7, 2017 at 9:41 PM Hen  wrote:

> Yes. You would just do it quicker (if such was truly needed). You would
> still need at least 3 +1s, and at the moment there is the crutch that you
> would also need 3 Incubator PMC +1s (or the first 3 would have to be IPMC).
>
> If it's a security vulnerability, then we would use private@ and confer
> with security@ on the process.
>
> Hen
>
> On Tue, Nov 7, 2017 at 1:45 PM, Chris Olivier 
> wrote:
>
> > Do we need a vote for a critical patch release?
> >
> > On Tue, Nov 7, 2017 at 12:08 PM, Meghna Baijal <
> meghnabaijal2...@gmail.com
> > >
> > wrote:
> >
> > > Hi All,
> > > A patch release is being planned for Apache MXNet(incubating) to
> include
> > > some important bug fixes.
> > > I want to confirm that for a patch release, there is no RC and voting
> > step.
> > >
> > > Effectively the process looks something like this -
> > >
> > > Step 1: Cherrypick all the necessary bugfixes into the existing release
> > > branch.
> > >
> > > Step 2: Add a PR to update the version number etc (Example
> > > https://github.com/apache/incubator-mxnet/commit/
> > > e0c7906693f0c79b0ce34a4d777c26a6bf1903c1
> > > )
> > >
> > > Step 3: Make sure the release branch build passes on
> > builds.apache.jenkins
> > > and nightly tests
> > >
> > > Step 4: tag the release branch in GitHub with the release tag - example
> > > 0.12.1
> > >
> > > Step 5: Download from the repo and checkout the tag, package it and
> Sign
> > > the src code into 'apache-mxnet-src-0.12.1-incubating'
> > >
> > > Step 6: Validate the signatures and test
> > >
> > > Step 7: Update the website. (pip and docker release)
> > >
> > > Step 8: Upload the src tar to the final destination -
> > > http://www.apache.org/dist/incubator/mxnet/
> > >
> > > Step 9: Announce the patch release
> > >
> > > Thanks,
> > > Meghna Baijal
> > >
> >
>