Re: [DISCUSS] 1.5.0 Release Plan

[Lai Wei]

Hi dev@,

Here are the updated release tracker and the timeline for 1.5.0 release:
https://cwiki.apache.org/confluence/display/MXNET/1.5.0+Release+Plan+and+Status
We are working on 2 last blockers on nightly test failure[1][2] and would
like to fix them by 06/04 so we can start to tag 1.5.0

For [1] I am in favor of disabling the test if we can't fix in time,
similar to what we did on CPU failure in [4].
Large tensor tests require a lot of memory and we don't have a good
solution on what's the best way to test them yet. [5]

For [2], it should be fixed by [3]

[1] large tensor nightly GPU failure:
https://github.com/apache/incubator-mxnet/issues/14981
[2] AMP tutorial nightly GPU failure:
https://github.com/apache/incubator-mxnet/issues/15028
[3] nightly fix: https://github.com/apache/incubator-mxnet/pull/15141
[4] large tensor nightly CPU:
https://github.com/apache/incubator-mxnet/issues/14980
[5] testability of large tensor discussion:
https://lists.apache.org/thread.html/d7f397c3f32196cb66ad9deae55dcf9a06dda56b37cbb0399ea1799f@%3Cdev.mxnet.apache.org%3E

Any help will be appreciated, thanks a lot!


Best Regards

Lai


On Fri, May 31, 2019 at 2:31 PM Haibin Lin  wrote:

> Hi dev@,
>
> Quick update on the gluonnlp issue. Lai and I worked together to test
> gluonnlp and MXNet with different configurations, and found that the use of
> GELU operator in fp16 is causing the divergence. It was a very recent
> change in gluonnlp, and it can be avoided by reverting the change in
> GluonNLP. This doesn't block 1.5 release anymore.
>
> Best,
> Haibin
>
> On Thu, May 30, 2019 at 11:33 AM Lai Wei  wrote:
>
> > Hi dev@,
> >
> > Quick update on the 1.5.0 release, all previous tracked PRs have been
> > merged and CI is back to normal again, please rebase your PR.
> > Again, I would like to encourage downstream projects to test against
> latest
> > MXNet now to discover bugs and regressions early, really appreciate your
> > help.
> >
> > We still have 3 new open issues/PRs to track:
> > 1. Gluon NLP BERT training Haibin mentioned
> > 2. https://github.com/apache/incubator-mxnet/pull/15039
> > 3. https://github.com/apache/incubator-mxnet/pull/15097
> >
> > Thanks!
> >
> > Best Regards
> >
> > Lai
> >
> >
> > On Tue, May 28, 2019 at 9:32 AM Haibin Lin 
> > wrote:
> >
> > > Hi dev@,
> > >
> > > I was testing GluonNLP with MXNet master, and found that BERT training
> > > crashes a few hours after I launch the job. I can confirm that MXNet
> pip
> > > package 20190412 works fine. I am bisecting changes in MXNet/GluonNLP
> to
> > > check what causes the problem. I'll send an update as soon as I find
> the
> > > root cause, or if I find any workaround.
> > >
> > > Thanks,
> > > Haibin
> > >
> > > On Thu, May 23, 2019 at 2:12 AM Lin Yuan  wrote:
> > >
> > > > Hi Lai,
> > > >
> > > > One important PR that is currently blocked by a Flaky TensorRT test:
> > > >
> > > > https://github.com/apache/incubator-mxnet/pull/15041
> > > >
> > > > I have retriggered it several times. If it fails again, I may need CI
> > > team
> > > > to help disable this test. It has been reported by multiple people:
> > > > https://github.com/apache/incubator-mxnet/issues/14978
> > > >
> > > > Thanks,
> > > >
> > > > Lin
> > > >
> > > > On Wed, May 22, 2019 at 11:38 PM Zhao, Patric  >
> > > > wrote:
> > > >
> > > > > Thanks, Lai.
> > > > >
> > > > > With the great helps from the community, all PRs listed in the
> > roadmap
> > > > are
> > > > > done :)
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-mxnet/issues/14619#issuecomment-480110642
> > > > >
> > > > > Update the status of the below list
> > > > >
> > > > >  - [1] PR#14713 is almost done and wait for internal validation
> > results
> > > > >  - [2] PR#14893 is merged
> > > > >  - [3] PR#15031 is merged
> > > > >  - [7] PR#15038 new PR to fix the bug in C++ interface, will be
> > merged
> > > > > soon after the review.
> > > > >
> > > > > Feel free to let me know if anything our team can help :)
> > > > >
> > > > > BR,
> > > > >
> > > > > --Patric
> > > > >
> > > > > > 

Re: [DISCUSS] 1.5.0 Release Plan

[Haibin Lin]

Hi dev@,

Quick update on the gluonnlp issue. Lai and I worked together to test
gluonnlp and MXNet with different configurations, and found that the use of
GELU operator in fp16 is causing the divergence. It was a very recent
change in gluonnlp, and it can be avoided by reverting the change in
GluonNLP. This doesn't block 1.5 release anymore.

Best,
Haibin

On Thu, May 30, 2019 at 11:33 AM Lai Wei  wrote:

> Hi dev@,
>
> Quick update on the 1.5.0 release, all previous tracked PRs have been
> merged and CI is back to normal again, please rebase your PR.
> Again, I would like to encourage downstream projects to test against latest
> MXNet now to discover bugs and regressions early, really appreciate your
> help.
>
> We still have 3 new open issues/PRs to track:
> 1. Gluon NLP BERT training Haibin mentioned
> 2. https://github.com/apache/incubator-mxnet/pull/15039
> 3. https://github.com/apache/incubator-mxnet/pull/15097
>
> Thanks!
>
> Best Regards
>
> Lai
>
>
> On Tue, May 28, 2019 at 9:32 AM Haibin Lin 
> wrote:
>
> > Hi dev@,
> >
> > I was testing GluonNLP with MXNet master, and found that BERT training
> > crashes a few hours after I launch the job. I can confirm that MXNet pip
> > package 20190412 works fine. I am bisecting changes in MXNet/GluonNLP to
> > check what causes the problem. I'll send an update as soon as I find the
> > root cause, or if I find any workaround.
> >
> > Thanks,
> > Haibin
> >
> > On Thu, May 23, 2019 at 2:12 AM Lin Yuan  wrote:
> >
> > > Hi Lai,
> > >
> > > One important PR that is currently blocked by a Flaky TensorRT test:
> > >
> > > https://github.com/apache/incubator-mxnet/pull/15041
> > >
> > > I have retriggered it several times. If it fails again, I may need CI
> > team
> > > to help disable this test. It has been reported by multiple people:
> > > https://github.com/apache/incubator-mxnet/issues/14978
> > >
> > > Thanks,
> > >
> > > Lin
> > >
> > > On Wed, May 22, 2019 at 11:38 PM Zhao, Patric 
> > > wrote:
> > >
> > > > Thanks, Lai.
> > > >
> > > > With the great helps from the community, all PRs listed in the
> roadmap
> > > are
> > > > done :)
> > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-mxnet/issues/14619#issuecomment-480110642
> > > >
> > > > Update the status of the below list
> > > >
> > > >  - [1] PR#14713 is almost done and wait for internal validation
> results
> > > >  - [2] PR#14893 is merged
> > > >  - [3] PR#15031 is merged
> > > >  - [7] PR#15038 new PR to fix the bug in C++ interface, will be
> merged
> > > > soon after the review.
> > > >
> > > > Feel free to let me know if anything our team can help :)
> > > >
> > > > BR,
> > > >
> > > > --Patric
> > > >
> > > > > -Original Message-
> > > > > From: Lai Wei [mailto:roywei...@gmail.com]
> > > > > Sent: Thursday, May 23, 2019 6:05 AM
> > > > > To: dev@mxnet.incubator.apache.org
> > > > > Subject: Re: [DISCUSS] 1.5.0 Release Plan
> > > > >
> > > > > Hi @dev,
> > > > >
> > > > > Thanks for working hard for the 1.5 release, since there has been
> > > several
> > > > > release blockers (mostly fixed). We are extending the code freeze
> to
> > > > Friday
> > > > > 05/22/2019. Right now we are tracking the following 5 open
> > > > PRs[1][2][3][4][5]
> > > > > and 1 issue[6]. Please let us know if you need more time.
> > > > >
> > > > > I would like to encourage all downstream projects to test with
> latest
> > > > MXNet
> > > > > to avoid any incompatibility in the coming 1.5.0 release. If you
> have
> > > any
> > > > > issues that may block the release, please let us know.
> > > > > Thank you very much.
> > > > >
> > > > > [1] https://github.com/apache/incubator-mxnet/pull/14713
> > > > > [2] https://github.com/apache/incubator-mxnet/pull/14893
> > > > > [3] https://github.com/apache/incubator-mxnet/pull/15031
> > > > > [4] https://github.com/apache/incubator-mxnet/pull/15039
> > > > > [5] https://github.com/apache/incubator-mxnet/pull/15041
> > > > > [6] https://github.com/apa

Re: [DISCUSS] 1.5.0 Release Plan

[Lai Wei]

Hi dev@,

Quick update on the 1.5.0 release, all previous tracked PRs have been
merged and CI is back to normal again, please rebase your PR.
Again, I would like to encourage downstream projects to test against latest
MXNet now to discover bugs and regressions early, really appreciate your
help.

We still have 3 new open issues/PRs to track:
1. Gluon NLP BERT training Haibin mentioned
2. https://github.com/apache/incubator-mxnet/pull/15039
3. https://github.com/apache/incubator-mxnet/pull/15097

Thanks!

Best Regards

Lai


On Tue, May 28, 2019 at 9:32 AM Haibin Lin  wrote:

> Hi dev@,
>
> I was testing GluonNLP with MXNet master, and found that BERT training
> crashes a few hours after I launch the job. I can confirm that MXNet pip
> package 20190412 works fine. I am bisecting changes in MXNet/GluonNLP to
> check what causes the problem. I'll send an update as soon as I find the
> root cause, or if I find any workaround.
>
> Thanks,
> Haibin
>
> On Thu, May 23, 2019 at 2:12 AM Lin Yuan  wrote:
>
> > Hi Lai,
> >
> > One important PR that is currently blocked by a Flaky TensorRT test:
> >
> > https://github.com/apache/incubator-mxnet/pull/15041
> >
> > I have retriggered it several times. If it fails again, I may need CI
> team
> > to help disable this test. It has been reported by multiple people:
> > https://github.com/apache/incubator-mxnet/issues/14978
> >
> > Thanks,
> >
> > Lin
> >
> > On Wed, May 22, 2019 at 11:38 PM Zhao, Patric 
> > wrote:
> >
> > > Thanks, Lai.
> > >
> > > With the great helps from the community, all PRs listed in the roadmap
> > are
> > > done :)
> > >
> > >
> >
> https://github.com/apache/incubator-mxnet/issues/14619#issuecomment-480110642
> > >
> > > Update the status of the below list
> > >
> > >  - [1] PR#14713 is almost done and wait for internal validation results
> > >  - [2] PR#14893 is merged
> > >  - [3] PR#15031 is merged
> > >  - [7] PR#15038 new PR to fix the bug in C++ interface, will be merged
> > > soon after the review.
> > >
> > > Feel free to let me know if anything our team can help :)
> > >
> > > BR,
> > >
> > > --Patric
> > >
> > > > -Original Message-
> > > > From: Lai Wei [mailto:roywei...@gmail.com]
> > > > Sent: Thursday, May 23, 2019 6:05 AM
> > > > To: dev@mxnet.incubator.apache.org
> > > > Subject: Re: [DISCUSS] 1.5.0 Release Plan
> > > >
> > > > Hi @dev,
> > > >
> > > > Thanks for working hard for the 1.5 release, since there has been
> > several
> > > > release blockers (mostly fixed). We are extending the code freeze to
> > > Friday
> > > > 05/22/2019. Right now we are tracking the following 5 open
> > > PRs[1][2][3][4][5]
> > > > and 1 issue[6]. Please let us know if you need more time.
> > > >
> > > > I would like to encourage all downstream projects to test with latest
> > > MXNet
> > > > to avoid any incompatibility in the coming 1.5.0 release. If you have
> > any
> > > > issues that may block the release, please let us know.
> > > > Thank you very much.
> > > >
> > > > [1] https://github.com/apache/incubator-mxnet/pull/14713
> > > > [2] https://github.com/apache/incubator-mxnet/pull/14893
> > > > [3] https://github.com/apache/incubator-mxnet/pull/15031
> > > > [4] https://github.com/apache/incubator-mxnet/pull/15039
> > > > [5] https://github.com/apache/incubator-mxnet/pull/15041
> > > > [6] https://github.com/apache/incubator-mxnet/issues/15034
> > > >
> > > >
> > > > Best Regards
> > > >
> > > > Lai
> > > >
> > > >
> > > > On Wed, May 15, 2019 at 9:05 PM Junru Shao 
> > > > wrote:
> > > >
> > > > > Hi folks,
> > > > >
> > > > > Here I may have a release blocker for 1.5.0 about implementation of
> > > > > dynamic shape mechanism, which somehow conflicts with Gluon's
> > > > deferred
> > > > > initialization [1].
> > > > >
> > > > > [1] https://github.com/dmlc/gluon-nlp/issues/706
> > > > >
> > > > > On Wed, May 15, 2019 at 12:09 PM Anirudh Subramanian <
> > > > > anirudh2...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Lai,
> >

Re: [DISCUSS] 1.5.0 Release Plan

[Haibin Lin]

Hi dev@,

I was testing GluonNLP with MXNet master, and found that BERT training
crashes a few hours after I launch the job. I can confirm that MXNet pip
package 20190412 works fine. I am bisecting changes in MXNet/GluonNLP to
check what causes the problem. I'll send an update as soon as I find the
root cause, or if I find any workaround.

Thanks,
Haibin

On Thu, May 23, 2019 at 2:12 AM Lin Yuan  wrote:

> Hi Lai,
>
> One important PR that is currently blocked by a Flaky TensorRT test:
>
> https://github.com/apache/incubator-mxnet/pull/15041
>
> I have retriggered it several times. If it fails again, I may need CI team
> to help disable this test. It has been reported by multiple people:
> https://github.com/apache/incubator-mxnet/issues/14978
>
> Thanks,
>
> Lin
>
> On Wed, May 22, 2019 at 11:38 PM Zhao, Patric 
> wrote:
>
> > Thanks, Lai.
> >
> > With the great helps from the community, all PRs listed in the roadmap
> are
> > done :)
> >
> >
> https://github.com/apache/incubator-mxnet/issues/14619#issuecomment-480110642
> >
> > Update the status of the below list
> >
> >  - [1] PR#14713 is almost done and wait for internal validation results
> >  - [2] PR#14893 is merged
> >  - [3] PR#15031 is merged
> >  - [7] PR#15038 new PR to fix the bug in C++ interface, will be merged
> > soon after the review.
> >
> > Feel free to let me know if anything our team can help :)
> >
> > BR,
> >
> > --Patric
> >
> > > -----Original Message-
> > > From: Lai Wei [mailto:roywei...@gmail.com]
> > > Sent: Thursday, May 23, 2019 6:05 AM
> > > To: dev@mxnet.incubator.apache.org
> > > Subject: Re: [DISCUSS] 1.5.0 Release Plan
> > >
> > > Hi @dev,
> > >
> > > Thanks for working hard for the 1.5 release, since there has been
> several
> > > release blockers (mostly fixed). We are extending the code freeze to
> > Friday
> > > 05/22/2019. Right now we are tracking the following 5 open
> > PRs[1][2][3][4][5]
> > > and 1 issue[6]. Please let us know if you need more time.
> > >
> > > I would like to encourage all downstream projects to test with latest
> > MXNet
> > > to avoid any incompatibility in the coming 1.5.0 release. If you have
> any
> > > issues that may block the release, please let us know.
> > > Thank you very much.
> > >
> > > [1] https://github.com/apache/incubator-mxnet/pull/14713
> > > [2] https://github.com/apache/incubator-mxnet/pull/14893
> > > [3] https://github.com/apache/incubator-mxnet/pull/15031
> > > [4] https://github.com/apache/incubator-mxnet/pull/15039
> > > [5] https://github.com/apache/incubator-mxnet/pull/15041
> > > [6] https://github.com/apache/incubator-mxnet/issues/15034
> > >
> > >
> > > Best Regards
> > >
> > > Lai
> > >
> > >
> > > On Wed, May 15, 2019 at 9:05 PM Junru Shao 
> > > wrote:
> > >
> > > > Hi folks,
> > > >
> > > > Here I may have a release blocker for 1.5.0 about implementation of
> > > > dynamic shape mechanism, which somehow conflicts with Gluon's
> > > deferred
> > > > initialization [1].
> > > >
> > > > [1] https://github.com/dmlc/gluon-nlp/issues/706
> > > >
> > > > On Wed, May 15, 2019 at 12:09 PM Anirudh Subramanian <
> > > > anirudh2...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Lai,
> > > > >
> > > > > From the discussion I had with Nvidia offline they are targeting on
> > > > pushing
> > > > > the required changes today.
> > > > > Since this is important feature for the release, if this gets
> > > > > delayed and cannot  be merged by 05/17/2019, the code freeze date
> > > > > may need to be changed.
> > > > >
> > > > > Anirudh
> > > > >
> > > > > On Wed, May 15, 2019 at 1:23 AM Lv, Tao A 
> > wrote:
> > > > >
> > > > > > Hi dev,
> > > > > >
> > > > > > We see there are several github issues [1][2][3][4] about mxnet
> > > > > > windows build experience. The team is working intensively
> > > > > > [5][6][7] on that to
> > > > > fix
> > > > > > some problems of MKL-DNN build on windows. We hope these fixes
> > > can
> > > > catch
> > > > > > the code 

Re: [DISCUSS] 1.5.0 Release Plan

[Lin Yuan]

Hi Lai,

One important PR that is currently blocked by a Flaky TensorRT test:

https://github.com/apache/incubator-mxnet/pull/15041

I have retriggered it several times. If it fails again, I may need CI team
to help disable this test. It has been reported by multiple people:
https://github.com/apache/incubator-mxnet/issues/14978

Thanks,

Lin

On Wed, May 22, 2019 at 11:38 PM Zhao, Patric  wrote:

> Thanks, Lai.
>
> With the great helps from the community, all PRs listed in the roadmap are
> done :)
>
> https://github.com/apache/incubator-mxnet/issues/14619#issuecomment-480110642
>
> Update the status of the below list
>
>  - [1] PR#14713 is almost done and wait for internal validation results
>  - [2] PR#14893 is merged
>  - [3] PR#15031 is merged
>  - [7] PR#15038 new PR to fix the bug in C++ interface, will be merged
> soon after the review.
>
> Feel free to let me know if anything our team can help :)
>
> BR,
>
> --Patric
>
> > -Original Message-
> > From: Lai Wei [mailto:roywei...@gmail.com]
> > Sent: Thursday, May 23, 2019 6:05 AM
> > To: dev@mxnet.incubator.apache.org
> > Subject: Re: [DISCUSS] 1.5.0 Release Plan
> >
> > Hi @dev,
> >
> > Thanks for working hard for the 1.5 release, since there has been several
> > release blockers (mostly fixed). We are extending the code freeze to
> Friday
> > 05/22/2019. Right now we are tracking the following 5 open
> PRs[1][2][3][4][5]
> > and 1 issue[6]. Please let us know if you need more time.
> >
> > I would like to encourage all downstream projects to test with latest
> MXNet
> > to avoid any incompatibility in the coming 1.5.0 release. If you have any
> > issues that may block the release, please let us know.
> > Thank you very much.
> >
> > [1] https://github.com/apache/incubator-mxnet/pull/14713
> > [2] https://github.com/apache/incubator-mxnet/pull/14893
> > [3] https://github.com/apache/incubator-mxnet/pull/15031
> > [4] https://github.com/apache/incubator-mxnet/pull/15039
> > [5] https://github.com/apache/incubator-mxnet/pull/15041
> > [6] https://github.com/apache/incubator-mxnet/issues/15034
> >
> >
> > Best Regards
> >
> > Lai
> >
> >
> > On Wed, May 15, 2019 at 9:05 PM Junru Shao 
> > wrote:
> >
> > > Hi folks,
> > >
> > > Here I may have a release blocker for 1.5.0 about implementation of
> > > dynamic shape mechanism, which somehow conflicts with Gluon's
> > deferred
> > > initialization [1].
> > >
> > > [1] https://github.com/dmlc/gluon-nlp/issues/706
> > >
> > > On Wed, May 15, 2019 at 12:09 PM Anirudh Subramanian <
> > > anirudh2...@gmail.com>
> > > wrote:
> > >
> > > > Hi Lai,
> > > >
> > > > From the discussion I had with Nvidia offline they are targeting on
> > > pushing
> > > > the required changes today.
> > > > Since this is important feature for the release, if this gets
> > > > delayed and cannot  be merged by 05/17/2019, the code freeze date
> > > > may need to be changed.
> > > >
> > > > Anirudh
> > > >
> > > > On Wed, May 15, 2019 at 1:23 AM Lv, Tao A 
> wrote:
> > > >
> > > > > Hi dev,
> > > > >
> > > > > We see there are several github issues [1][2][3][4] about mxnet
> > > > > windows build experience. The team is working intensively
> > > > > [5][6][7] on that to
> > > > fix
> > > > > some problems of MKL-DNN build on windows. We hope these fixes
> > can
> > > catch
> > > > > the code freeze and finally enter the 1.5.0 release.
> > > > >
> > > > > The PR against mshadow (#374) was already merged and MXNet PR
> > > > > #14877 is under review - great thanks to CI team for helping on
> > > > > the MKL
> > > > installation
> > > > > request. PR #14952 is document change according to build logic
> > > > > changes
> > > in
> > > > > PR #14877. So I think these two PRs should be merged
> simultaneously.
> > > > > Currently #14877 is experiencing a CI response problem.
> > > > >
> > > > > Please take your time to have a look at these two PRs. Your
> > > > > comments
> > > and
> > > > > suggestions are highly appreciated.
> > > > >
> > > > > Thanks,
> > > > > -tao
> > > > >
> > 

RE: [DISCUSS] 1.5.0 Release Plan

[Zhao, Patric]

Thanks, Lai.  

With the great helps from the community, all PRs listed in the roadmap are done 
:)
https://github.com/apache/incubator-mxnet/issues/14619#issuecomment-480110642

Update the status of the below list

 - [1] PR#14713 is almost done and wait for internal validation results
 - [2] PR#14893 is merged
 - [3] PR#15031 is merged
 - [7] PR#15038 new PR to fix the bug in C++ interface, will be merged soon 
after the review.

Feel free to let me know if anything our team can help :)

BR,

--Patric

> -Original Message-
> From: Lai Wei [mailto:roywei...@gmail.com]
> Sent: Thursday, May 23, 2019 6:05 AM
> To: dev@mxnet.incubator.apache.org
> Subject: Re: [DISCUSS] 1.5.0 Release Plan
> 
> Hi @dev,
> 
> Thanks for working hard for the 1.5 release, since there has been several
> release blockers (mostly fixed). We are extending the code freeze to Friday
> 05/22/2019. Right now we are tracking the following 5 open PRs[1][2][3][4][5]
> and 1 issue[6]. Please let us know if you need more time.
> 
> I would like to encourage all downstream projects to test with latest MXNet
> to avoid any incompatibility in the coming 1.5.0 release. If you have any
> issues that may block the release, please let us know.
> Thank you very much.
> 
> [1] https://github.com/apache/incubator-mxnet/pull/14713
> [2] https://github.com/apache/incubator-mxnet/pull/14893
> [3] https://github.com/apache/incubator-mxnet/pull/15031
> [4] https://github.com/apache/incubator-mxnet/pull/15039
> [5] https://github.com/apache/incubator-mxnet/pull/15041
> [6] https://github.com/apache/incubator-mxnet/issues/15034
> 
> 
> Best Regards
> 
> Lai
> 
> 
> On Wed, May 15, 2019 at 9:05 PM Junru Shao 
> wrote:
> 
> > Hi folks,
> >
> > Here I may have a release blocker for 1.5.0 about implementation of
> > dynamic shape mechanism, which somehow conflicts with Gluon's
> deferred
> > initialization [1].
> >
> > [1] https://github.com/dmlc/gluon-nlp/issues/706
> >
> > On Wed, May 15, 2019 at 12:09 PM Anirudh Subramanian <
> > anirudh2...@gmail.com>
> > wrote:
> >
> > > Hi Lai,
> > >
> > > From the discussion I had with Nvidia offline they are targeting on
> > pushing
> > > the required changes today.
> > > Since this is important feature for the release, if this gets
> > > delayed and cannot  be merged by 05/17/2019, the code freeze date
> > > may need to be changed.
> > >
> > > Anirudh
> > >
> > > On Wed, May 15, 2019 at 1:23 AM Lv, Tao A  wrote:
> > >
> > > > Hi dev,
> > > >
> > > > We see there are several github issues [1][2][3][4] about mxnet
> > > > windows build experience. The team is working intensively
> > > > [5][6][7] on that to
> > > fix
> > > > some problems of MKL-DNN build on windows. We hope these fixes
> can
> > catch
> > > > the code freeze and finally enter the 1.5.0 release.
> > > >
> > > > The PR against mshadow (#374) was already merged and MXNet PR
> > > > #14877 is under review - great thanks to CI team for helping on
> > > > the MKL
> > > installation
> > > > request. PR #14952 is document change according to build logic
> > > > changes
> > in
> > > > PR #14877. So I think these two PRs should be merged simultaneously.
> > > > Currently #14877 is experiencing a CI response problem.
> > > >
> > > > Please take your time to have a look at these two PRs. Your
> > > > comments
> > and
> > > > suggestions are highly appreciated.
> > > >
> > > > Thanks,
> > > > -tao
> > > >
> > > > [1] https://github.com/apache/incubator-mxnet/issues/14670
> > > > [2] https://github.com/apache/incubator-mxnet/issues/14335
> > > > [3] https://github.com/apache/incubator-mxnet/issues/14203
> > > > [4] https://github.com/apache/incubator-mxnet/issues/14085
> > > > [5] https://github.com/apache/incubator-mxnet/pull/14877
> > > > [6] https://github.com/dmlc/mshadow/pull/374
> > > > [7] https://github.com/apache/incubator-mxnet/pull/14952
> > > >
> > > > -Original Message-
> > > > From: Lai Wei [mailto:roywei...@gmail.com]
> > > > Sent: Wednesday, May 15, 2019 2:57 PM
> > > > To: dev@mxnet.incubator.apache.org
> > > > Subject: Re: [DISCUSS] 1.5.0 Release Plan
> > > >
> > > > Hi Anirudh,
> > > >
> > > > I see there was an offline disu

Re: [DISCUSS] 1.5.0 Release Plan

[Lai Wei]

Hi @dev,

Thanks for working hard for the 1.5 release, since there has been several
release blockers (mostly fixed). We are extending the code freeze to Friday
05/22/2019. Right now we are tracking the following 5 open
PRs[1][2][3][4][5] and 1 issue[6]. Please let us know if you need more
time.

I would like to encourage all downstream projects to test with latest MXNet
to avoid any incompatibility in the coming 1.5.0 release. If you have any
issues that may block the release, please let us know.
Thank you very much.

[1] https://github.com/apache/incubator-mxnet/pull/14713
[2] https://github.com/apache/incubator-mxnet/pull/14893
[3] https://github.com/apache/incubator-mxnet/pull/15031
[4] https://github.com/apache/incubator-mxnet/pull/15039
[5] https://github.com/apache/incubator-mxnet/pull/15041
[6] https://github.com/apache/incubator-mxnet/issues/15034


Best Regards

Lai


On Wed, May 15, 2019 at 9:05 PM Junru Shao  wrote:

> Hi folks,
>
> Here I may have a release blocker for 1.5.0 about implementation of dynamic
> shape mechanism, which somehow conflicts with Gluon's deferred
> initialization [1].
>
> [1] https://github.com/dmlc/gluon-nlp/issues/706
>
> On Wed, May 15, 2019 at 12:09 PM Anirudh Subramanian <
> anirudh2...@gmail.com>
> wrote:
>
> > Hi Lai,
> >
> > From the discussion I had with Nvidia offline they are targeting on
> pushing
> > the required changes today.
> > Since this is important feature for the release, if this gets delayed and
> > cannot  be merged by 05/17/2019,
> > the code freeze date may need to be changed.
> >
> > Anirudh
> >
> > On Wed, May 15, 2019 at 1:23 AM Lv, Tao A  wrote:
> >
> > > Hi dev,
> > >
> > > We see there are several github issues [1][2][3][4] about mxnet windows
> > > build experience. The team is working intensively [5][6][7] on that to
> > fix
> > > some problems of MKL-DNN build on windows. We hope these fixes can
> catch
> > > the code freeze and finally enter the 1.5.0 release.
> > >
> > > The PR against mshadow (#374) was already merged and MXNet PR #14877 is
> > > under review - great thanks to CI team for helping on the MKL
> > installation
> > > request. PR #14952 is document change according to build logic changes
> in
> > > PR #14877. So I think these two PRs should be merged simultaneously.
> > > Currently #14877 is experiencing a CI response problem.
> > >
> > > Please take your time to have a look at these two PRs. Your comments
> and
> > > suggestions are highly appreciated.
> > >
> > > Thanks,
> > > -tao
> > >
> > > [1] https://github.com/apache/incubator-mxnet/issues/14670
> > > [2] https://github.com/apache/incubator-mxnet/issues/14335
> > > [3] https://github.com/apache/incubator-mxnet/issues/14203
> > > [4] https://github.com/apache/incubator-mxnet/issues/14085
> > > [5] https://github.com/apache/incubator-mxnet/pull/14877
> > > [6] https://github.com/dmlc/mshadow/pull/374
> > > [7] https://github.com/apache/incubator-mxnet/pull/14952
> > >
> > > -Original Message-
> > > From: Lai Wei [mailto:roywei...@gmail.com]
> > > Sent: Wednesday, May 15, 2019 2:57 PM
> > > To: dev@mxnet.incubator.apache.org
> > > Subject: Re: [DISCUSS] 1.5.0 Release Plan
> > >
> > > Hi Anirudh,
> > >
> > > I see there was an offline disucssion
> > > <
> > >
> >
> https://github.com/apache/incubator-mxnet/pull/14173#pullrequestreview-235846341
> > > >
> > > and I have updated the AMP feature and your project on the release
> > tracker
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/MXNET/1.5.0+Release+Plan+and+Status
> > > >
> > > ,
> > > Please let me know if you have any updates.
> > >
> > > Hi @dev,
> > > This is a gentle reminder that  the code freeze for 1.5.0 release is on
> > > 05/17/2019, please let us know if you have any WIP pull requests aiming
> > for
> > > 1.5.0 that needs attention.
> > > Please understand we already have around 650 commits in master that
> need
> > > to be released in time. We understand TensorRT test in CI is failing
> and
> > > are trying to fix it. Meanwhile please update the tracker if there is
> any
> > > change:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/MXNET/1.5.0+Release+Plan+and+Status
> > >
> > > Thanks!
> > >
> > > Lai
> > >
> > >
> 

Re: [DISCUSS] 1.5.0 Release Plan

[Junru Shao]

Hi folks,

Here I may have a release blocker for 1.5.0 about implementation of dynamic
shape mechanism, which somehow conflicts with Gluon's deferred
initialization [1].

[1] https://github.com/dmlc/gluon-nlp/issues/706

On Wed, May 15, 2019 at 12:09 PM Anirudh Subramanian 
wrote:

> Hi Lai,
>
> From the discussion I had with Nvidia offline they are targeting on pushing
> the required changes today.
> Since this is important feature for the release, if this gets delayed and
> cannot  be merged by 05/17/2019,
> the code freeze date may need to be changed.
>
> Anirudh
>
> On Wed, May 15, 2019 at 1:23 AM Lv, Tao A  wrote:
>
> > Hi dev,
> >
> > We see there are several github issues [1][2][3][4] about mxnet windows
> > build experience. The team is working intensively [5][6][7] on that to
> fix
> > some problems of MKL-DNN build on windows. We hope these fixes can catch
> > the code freeze and finally enter the 1.5.0 release.
> >
> > The PR against mshadow (#374) was already merged and MXNet PR #14877 is
> > under review - great thanks to CI team for helping on the MKL
> installation
> > request. PR #14952 is document change according to build logic changes in
> > PR #14877. So I think these two PRs should be merged simultaneously.
> > Currently #14877 is experiencing a CI response problem.
> >
> > Please take your time to have a look at these two PRs. Your comments and
> > suggestions are highly appreciated.
> >
> > Thanks,
> > -tao
> >
> > [1] https://github.com/apache/incubator-mxnet/issues/14670
> > [2] https://github.com/apache/incubator-mxnet/issues/14335
> > [3] https://github.com/apache/incubator-mxnet/issues/14203
> > [4] https://github.com/apache/incubator-mxnet/issues/14085
> > [5] https://github.com/apache/incubator-mxnet/pull/14877
> > [6] https://github.com/dmlc/mshadow/pull/374
> > [7] https://github.com/apache/incubator-mxnet/pull/14952
> >
> > -Original Message-
> > From: Lai Wei [mailto:roywei...@gmail.com]
> > Sent: Wednesday, May 15, 2019 2:57 PM
> > To: dev@mxnet.incubator.apache.org
> > Subject: Re: [DISCUSS] 1.5.0 Release Plan
> >
> > Hi Anirudh,
> >
> > I see there was an offline disucssion
> > <
> >
> https://github.com/apache/incubator-mxnet/pull/14173#pullrequestreview-235846341
> > >
> > and I have updated the AMP feature and your project on the release
> tracker
> > <
> >
> https://cwiki.apache.org/confluence/display/MXNET/1.5.0+Release+Plan+and+Status
> > >
> > ,
> > Please let me know if you have any updates.
> >
> > Hi @dev,
> > This is a gentle reminder that  the code freeze for 1.5.0 release is on
> > 05/17/2019, please let us know if you have any WIP pull requests aiming
> for
> > 1.5.0 that needs attention.
> > Please understand we already have around 650 commits in master that need
> > to be released in time. We understand TensorRT test in CI is failing and
> > are trying to fix it. Meanwhile please update the tracker if there is any
> > change:
> >
> >
> https://cwiki.apache.org/confluence/display/MXNET/1.5.0+Release+Plan+and+Status
> >
> > Thanks!
> >
> > Lai
> >
> >
> > On Wed, May 8, 2019 at 11:58 AM Anirudh Subramanian <
> anirudh2...@gmail.com
> > >
> > wrote:
> >
> > > Hi Sheng,
> > >
> > > I had a discussion with nvidia folks offline today (@ptrendx et. al.).
> > > I strongly feel that the AMP feature should be included as part of the
> > > release: https://github.com/apache/incubator-mxnet/pull/14173 .
> > > The PR is aimed for completion for next week but reviews and RFC
> > > discussions may take some time. I would request to extend the release
> > > code freeze by 2 weeks.
> > > Also, I would like to include
> > >
> > > https://cwiki.apache.org/confluence/display/MXNET/Conversion+from+FP32
> > > +to+Mixed+Precision+Models
> > > which
> > > depends on the AMP PR.
> > > I am also aiming for adding a PR by this week end or early next week,
> > > but reviews will take longer than May 17th.
> > >
> > > Anirudh
> > >
> > >
> > > On Mon, May 6, 2019 at 11:49 PM Sheng Zha  wrote:
> > >
> > > > Hi,
> > > >
> > > > While 1.4.1 vote on general@incubator is still on going, I’d like to
> > > > propose that we start preparing 1.5.0 release.
> > > >
> > > > 1.5.0 will include changes that dates back to last year and there
> 

Re: [DISCUSS] 1.5.0 Release Plan

[Anirudh Subramanian]

Hi Lai,

>From the discussion I had with Nvidia offline they are targeting on pushing
the required changes today.
Since this is important feature for the release, if this gets delayed and
cannot  be merged by 05/17/2019,
the code freeze date may need to be changed.

Anirudh

On Wed, May 15, 2019 at 1:23 AM Lv, Tao A  wrote:

> Hi dev,
>
> We see there are several github issues [1][2][3][4] about mxnet windows
> build experience. The team is working intensively [5][6][7] on that to fix
> some problems of MKL-DNN build on windows. We hope these fixes can catch
> the code freeze and finally enter the 1.5.0 release.
>
> The PR against mshadow (#374) was already merged and MXNet PR #14877 is
> under review - great thanks to CI team for helping on the MKL installation
> request. PR #14952 is document change according to build logic changes in
> PR #14877. So I think these two PRs should be merged simultaneously.
> Currently #14877 is experiencing a CI response problem.
>
> Please take your time to have a look at these two PRs. Your comments and
> suggestions are highly appreciated.
>
> Thanks,
> -tao
>
> [1] https://github.com/apache/incubator-mxnet/issues/14670
> [2] https://github.com/apache/incubator-mxnet/issues/14335
> [3] https://github.com/apache/incubator-mxnet/issues/14203
> [4] https://github.com/apache/incubator-mxnet/issues/14085
> [5] https://github.com/apache/incubator-mxnet/pull/14877
> [6] https://github.com/dmlc/mshadow/pull/374
> [7] https://github.com/apache/incubator-mxnet/pull/14952
>
> -Original Message-
> From: Lai Wei [mailto:roywei...@gmail.com]
> Sent: Wednesday, May 15, 2019 2:57 PM
> To: dev@mxnet.incubator.apache.org
> Subject: Re: [DISCUSS] 1.5.0 Release Plan
>
> Hi Anirudh,
>
> I see there was an offline disucssion
> <
> https://github.com/apache/incubator-mxnet/pull/14173#pullrequestreview-235846341
> >
> and I have updated the AMP feature and your project on the release tracker
> <
> https://cwiki.apache.org/confluence/display/MXNET/1.5.0+Release+Plan+and+Status
> >
> ,
> Please let me know if you have any updates.
>
> Hi @dev,
> This is a gentle reminder that  the code freeze for 1.5.0 release is on
> 05/17/2019, please let us know if you have any WIP pull requests aiming for
> 1.5.0 that needs attention.
> Please understand we already have around 650 commits in master that need
> to be released in time. We understand TensorRT test in CI is failing and
> are trying to fix it. Meanwhile please update the tracker if there is any
> change:
>
> https://cwiki.apache.org/confluence/display/MXNET/1.5.0+Release+Plan+and+Status
>
> Thanks!
>
> Lai
>
>
> On Wed, May 8, 2019 at 11:58 AM Anirudh Subramanian  >
> wrote:
>
> > Hi Sheng,
> >
> > I had a discussion with nvidia folks offline today (@ptrendx et. al.).
> > I strongly feel that the AMP feature should be included as part of the
> > release: https://github.com/apache/incubator-mxnet/pull/14173 .
> > The PR is aimed for completion for next week but reviews and RFC
> > discussions may take some time. I would request to extend the release
> > code freeze by 2 weeks.
> > Also, I would like to include
> >
> > https://cwiki.apache.org/confluence/display/MXNET/Conversion+from+FP32
> > +to+Mixed+Precision+Models
> > which
> > depends on the AMP PR.
> > I am also aiming for adding a PR by this week end or early next week,
> > but reviews will take longer than May 17th.
> >
> > Anirudh
> >
> >
> > On Mon, May 6, 2019 at 11:49 PM Sheng Zha  wrote:
> >
> > > Hi,
> > >
> > > While 1.4.1 vote on general@incubator is still on going, I’d like to
> > > propose that we start preparing 1.5.0 release.
> > >
> > > 1.5.0 will include changes that dates back to last year and there
> > > has
> > been
> > > a lot of new features and improvements in it, so it will likely time
> > > us more time to prepare than 1.4.1. I propose the following timeline:
> > > - Cut release branch: release branch already cut. Will sync with
> > > master branch on 5/15/2019 EOD.
> > > - Code freeze: 5/17/2019. No more changes unless the release branch
> > > is in a broken state.
> > > - Tag and vote: 5/20/2019 onward.
> > >
> > > Lai Wei (roywei@) expressed to me offline that he’s willing to help
> > drive
> > > this release as release manager, and I’m happy to help again as
> > committer.
> > >
> > > If you have features in progress that you’d like to include in 1.5.0:
> > > - Add your feature to the scope:
> > >
> > https://cwiki.apache.org/confluence/display/MXNET/1.5.0+Release+Plan+a
> > nd+Status
> > > - Indicate in this thread:
> > >   - how confident you are about making it happen before the code
> freeze.
> > > If not confident, provide estimate for a more manageable code freeze
> > > date so that people can discuss whether to extend the deadline or to
> > > skip one release for it.
> > > - whether your PR requires more attention to make it happen.
> > >
> > > Thanks for your attention. Comments and suggestions are also welcome.
> > >
> > > -sz
> >
>

RE: [DISCUSS] 1.5.0 Release Plan

[Lv, Tao A]

Hi dev,

We see there are several github issues [1][2][3][4] about mxnet windows build 
experience. The team is working intensively [5][6][7] on that to fix some 
problems of MKL-DNN build on windows. We hope these fixes can catch the code 
freeze and finally enter the 1.5.0 release.

The PR against mshadow (#374) was already merged and MXNet PR #14877 is under 
review - great thanks to CI team for helping on the MKL installation request. 
PR #14952 is document change according to build logic changes in PR #14877. So 
I think these two PRs should be merged simultaneously. Currently #14877 is 
experiencing a CI response problem.

Please take your time to have a look at these two PRs. Your comments and 
suggestions are highly appreciated.

Thanks,
-tao

[1] https://github.com/apache/incubator-mxnet/issues/14670 
[2] https://github.com/apache/incubator-mxnet/issues/14335  
[3] https://github.com/apache/incubator-mxnet/issues/14203 
[4] https://github.com/apache/incubator-mxnet/issues/14085  
[5] https://github.com/apache/incubator-mxnet/pull/14877 
[6] https://github.com/dmlc/mshadow/pull/374 
[7] https://github.com/apache/incubator-mxnet/pull/14952  

-Original Message-
From: Lai Wei [mailto:roywei...@gmail.com] 
Sent: Wednesday, May 15, 2019 2:57 PM
To: dev@mxnet.incubator.apache.org
Subject: Re: [DISCUSS] 1.5.0 Release Plan

Hi Anirudh,

I see there was an offline disucssion
<https://github.com/apache/incubator-mxnet/pull/14173#pullrequestreview-235846341>
and I have updated the AMP feature and your project on the release tracker 
<https://cwiki.apache.org/confluence/display/MXNET/1.5.0+Release+Plan+and+Status>
,
Please let me know if you have any updates.

Hi @dev,
This is a gentle reminder that  the code freeze for 1.5.0 release is on 
05/17/2019, please let us know if you have any WIP pull requests aiming for
1.5.0 that needs attention.
Please understand we already have around 650 commits in master that need to be 
released in time. We understand TensorRT test in CI is failing and are trying 
to fix it. Meanwhile please update the tracker if there is any
change:
https://cwiki.apache.org/confluence/display/MXNET/1.5.0+Release+Plan+and+Status

Thanks!

Lai


On Wed, May 8, 2019 at 11:58 AM Anirudh Subramanian 
wrote:

> Hi Sheng,
>
> I had a discussion with nvidia folks offline today (@ptrendx et. al.). 
> I strongly feel that the AMP feature should be included as part of the
> release: https://github.com/apache/incubator-mxnet/pull/14173 .
> The PR is aimed for completion for next week but reviews and RFC 
> discussions may take some time. I would request to extend the release 
> code freeze by 2 weeks.
> Also, I would like to include
>
> https://cwiki.apache.org/confluence/display/MXNET/Conversion+from+FP32
> +to+Mixed+Precision+Models
> which
> depends on the AMP PR.
> I am also aiming for adding a PR by this week end or early next week, 
> but reviews will take longer than May 17th.
>
> Anirudh
>
>
> On Mon, May 6, 2019 at 11:49 PM Sheng Zha  wrote:
>
> > Hi,
> >
> > While 1.4.1 vote on general@incubator is still on going, I’d like to 
> > propose that we start preparing 1.5.0 release.
> >
> > 1.5.0 will include changes that dates back to last year and there 
> > has
> been
> > a lot of new features and improvements in it, so it will likely time 
> > us more time to prepare than 1.4.1. I propose the following timeline:
> > - Cut release branch: release branch already cut. Will sync with 
> > master branch on 5/15/2019 EOD.
> > - Code freeze: 5/17/2019. No more changes unless the release branch 
> > is in a broken state.
> > - Tag and vote: 5/20/2019 onward.
> >
> > Lai Wei (roywei@) expressed to me offline that he’s willing to help
> drive
> > this release as release manager, and I’m happy to help again as
> committer.
> >
> > If you have features in progress that you’d like to include in 1.5.0:
> > - Add your feature to the scope:
> >
> https://cwiki.apache.org/confluence/display/MXNET/1.5.0+Release+Plan+a
> nd+Status
> > - Indicate in this thread:
> >   - how confident you are about making it happen before the code freeze.
> > If not confident, provide estimate for a more manageable code freeze 
> > date so that people can discuss whether to extend the deadline or to 
> > skip one release for it.
> > - whether your PR requires more attention to make it happen.
> >
> > Thanks for your attention. Comments and suggestions are also welcome.
> >
> > -sz
>

Re: [DISCUSS] 1.5.0 Release Plan

[Lai Wei]

Hi Anirudh,

I see there was an offline disucssion

and I have updated the AMP feature and your project on the release tracker

,
Please let me know if you have any updates.

Hi @dev,
This is a gentle reminder that  the code freeze for 1.5.0 release is on
05/17/2019, please let us know if you have any WIP pull requests aiming for
1.5.0 that needs attention.
Please understand we already have around 650 commits in master that need to
be released in time. We understand TensorRT test in CI is failing and are
trying to fix it. Meanwhile please update the tracker if there is any
change:
https://cwiki.apache.org/confluence/display/MXNET/1.5.0+Release+Plan+and+Status

Thanks!

Lai


On Wed, May 8, 2019 at 11:58 AM Anirudh Subramanian 
wrote:

> Hi Sheng,
>
> I had a discussion with nvidia folks offline today (@ptrendx et. al.). I
> strongly feel that the AMP feature should be included as part of the
> release: https://github.com/apache/incubator-mxnet/pull/14173 .
> The PR is aimed for completion for next week but reviews and RFC
> discussions may take some time. I would request to extend the release code
> freeze by 2 weeks.
> Also, I would like to include
>
> https://cwiki.apache.org/confluence/display/MXNET/Conversion+from+FP32+to+Mixed+Precision+Models
> which
> depends on the AMP PR.
> I am also aiming for adding a PR by this week end or early next week, but
> reviews will take longer than May 17th.
>
> Anirudh
>
>
> On Mon, May 6, 2019 at 11:49 PM Sheng Zha  wrote:
>
> > Hi,
> >
> > While 1.4.1 vote on general@incubator is still on going, I’d like to
> > propose that we start preparing 1.5.0 release.
> >
> > 1.5.0 will include changes that dates back to last year and there has
> been
> > a lot of new features and improvements in it, so it will likely time us
> > more time to prepare than 1.4.1. I propose the following timeline:
> > - Cut release branch: release branch already cut. Will sync with master
> > branch on 5/15/2019 EOD.
> > - Code freeze: 5/17/2019. No more changes unless the release branch is in
> > a broken state.
> > - Tag and vote: 5/20/2019 onward.
> >
> > Lai Wei (roywei@) expressed to me offline that he’s willing to help
> drive
> > this release as release manager, and I’m happy to help again as
> committer.
> >
> > If you have features in progress that you’d like to include in 1.5.0:
> > - Add your feature to the scope:
> >
> https://cwiki.apache.org/confluence/display/MXNET/1.5.0+Release+Plan+and+Status
> > - Indicate in this thread:
> >   - how confident you are about making it happen before the code freeze.
> > If not confident, provide estimate for a more manageable code freeze date
> > so that people can discuss whether to extend the deadline or to skip one
> > release for it.
> > - whether your PR requires more attention to make it happen.
> >
> > Thanks for your attention. Comments and suggestions are also welcome.
> >
> > -sz
>

Re: [DISCUSS] 1.5.0 Release Plan

[Anirudh Subramanian]

Hi Sheng,

I had a discussion with nvidia folks offline today (@ptrendx et. al.). I
strongly feel that the AMP feature should be included as part of the
release: https://github.com/apache/incubator-mxnet/pull/14173 .
The PR is aimed for completion for next week but reviews and RFC
discussions may take some time. I would request to extend the release code
freeze by 2 weeks.
Also, I would like to include
https://cwiki.apache.org/confluence/display/MXNET/Conversion+from+FP32+to+Mixed+Precision+Models
which
depends on the AMP PR.
I am also aiming for adding a PR by this week end or early next week, but
reviews will take longer than May 17th.

Anirudh


On Mon, May 6, 2019 at 11:49 PM Sheng Zha  wrote:

> Hi,
>
> While 1.4.1 vote on general@incubator is still on going, I’d like to
> propose that we start preparing 1.5.0 release.
>
> 1.5.0 will include changes that dates back to last year and there has been
> a lot of new features and improvements in it, so it will likely time us
> more time to prepare than 1.4.1. I propose the following timeline:
> - Cut release branch: release branch already cut. Will sync with master
> branch on 5/15/2019 EOD.
> - Code freeze: 5/17/2019. No more changes unless the release branch is in
> a broken state.
> - Tag and vote: 5/20/2019 onward.
>
> Lai Wei (roywei@) expressed to me offline that he’s willing to help drive
> this release as release manager, and I’m happy to help again as committer.
>
> If you have features in progress that you’d like to include in 1.5.0:
> - Add your feature to the scope:
> https://cwiki.apache.org/confluence/display/MXNET/1.5.0+Release+Plan+and+Status
> - Indicate in this thread:
>   - how confident you are about making it happen before the code freeze.
> If not confident, provide estimate for a more manageable code freeze date
> so that people can discuss whether to extend the deadline or to skip one
> release for it.
> - whether your PR requires more attention to make it happen.
>
> Thanks for your attention. Comments and suggestions are also welcome.
>
> -sz