When including the OMP fixes in 1.6, Chris's fix for a race condition should be included as well. So it's 3 PRs:
https://github.com/apache/incubator-mxnet/pull/17012 https://github.com/apache/incubator-mxnet/pull/17039 https://github.com/apache/incubator-mxnet/pull/17098 While all of these don't affect the binary Python builds that will be distributed for 1.6 release, they do affect any users building the 1.6 release from source with cmake. So it's beneficial to backport the 3 PRs. On Fri, 2019-12-27 at 11:24 -0800, Pedro Larroy wrote: > Agree with Sheng, I think it would be good to have the nice fixes that > Leonard has done for 1.6 and not delay them to further releases since they > are beneficial to users and developers. Thanks Leonard for helping fix > these long standing issues. > > On Fri, Dec 27, 2019 at 11:03 AM Lin Yuan <apefor...@gmail.com> wrote: > > > No, I just wanted to call it out because the title of the issue says > > "Failed > > OpenMP assertion when loading MXNet compiled with DEBUG=1 > > <https://github.com/apache/incubator-mxnet/issues/10856#>;". > > If this is considered a release blocker, I think we should backport it to > > 1.6. > > > > Thanks, > > Lin > > > > On Fri, Dec 27, 2019 at 10:47 AM Sheng Zha <szha....@gmail.com> wrote: > > > > > Reading these issues it’s pretty clear to me that these are fixes for > > > broken builds. I think we do consider broken builds to be release > > blockers. > > > Lin, am I missing something on which you base your suggestion for > > delaying > > > these changes? > > > > > > -sz > > > > > > > On Dec 27, 2019, at 10:30 AM, Lin Yuan <apefor...@gmail.com> wrote: > > > > > > > > Are these release blocker? It's very risky to make such last-minute > > big > > > > change after code freeze. > > > > > > > > Can we do this in the next release? > > > > > > > > Lin > > > > > > > > > On Fri, Dec 27, 2019 at 7:37 AM Lausen, Leonard > > > <lau...@amazon.com.invalid> > > > > > wrote: > > > > > > > > > > In case of backporting #17012, also > > > > > https://github.com/apache/incubator-mxnet/pull/17098 must be > > > backported. > > > > > The > > > > > updated OpenMP added a new target which is not used by MXNet but > > breaks > > > the > > > > > build on some systems with nvptx. #17098 disables building this unused > > > and > > > > > broken feature. > > > > > > > > > > > On Thu, 2019-12-26 at 12:55 -0800, Pedro Larroy wrote: > > > > > > https://github.com/apache/incubator-mxnet/pull/17012 should be also > > > > > ported > > > > > > to the release branch. > > > > > > > > > > > > On Fri, Dec 20, 2019 at 1:39 PM Przemysław Trędak < > > ptre...@apache.org> > > > > > > wrote: > > > > > > > > > > > > > That issue is now fixed in master, I am in the process of > > > > > cherry-picking > > > > > > > the fix to v1.6.x branch. I will prepare the RC1 once that is > > > > > > > ready. > > > > > > > > > > > > > > Thanks > > > > > > > Przemek > > > > > > > > > > > > > > On 2019/12/20 20:07:36, Lin Yuan <apefor...@gmail.com> wrote: > > > > > > > > What's the next step for the release? Should we continue testing > > > > > this and > > > > > > > > vote or wait until the > > > > > > > > https://github.com/apache/incubator-mxnet/issues/17105 is fixed? > > > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > > > Lin > > > > > > > > > > > > > > > > On Wed, Dec 18, 2019 at 12:55 AM Lausen, Leonard > > > > > > > <lau...@amazon.com.invalid> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Thanks Przemysław for managing this release and everyone who > > > > > > > contributed > > > > > > > > > to it. > > > > > > > > > > > > > > > > > > Unfortunately Zechen Wang just discovered another issue with > > > > > > > > > GPU > > > > > > > Pointwise > > > > > > > > > Fusion: https://github.com/apache/incubator-mxnet/issues/17105 > > > > > > > > > > > > > > > > > > Thus, -1. > > > > > > > > > > > > > > > > > > Unfortunately, as the nightly release pipeline was broken > > > > > > > > > until > > > > > > > recently > > > > > > > > > (and > > > > > > > > > still isn't re-set up completely yet), the issue hasn't been > > > > > discovered > > > > > > > > > earlier. > > > > > > > > > > > > > > > > > > Przemysław may have a quick fix for the issue. Another option > > > > > would be > > > > > > > to > > > > > > > > > release 1.6 with MXNET_USE_FUSION default to 0. > > > > > > > > > > > > > > > > > > Best regards > > > > > > > > > Leonard > > > > > > > > > > > > > > > > > > On Wed, 2019-12-18 at 05:30 +0000, Chen, Ciyong wrote: > > > > > > > > > > Appreciate Tredak to push out voting for 1.6 release. > > > > > > > > > > > > > > > > > > > > +1 as we've done lots of tests with expected performance in > > > > > > > > > > many > > > > > > > > > different > > > > > > > > > > scenarios including both single-node and multi-node (horovod > > > > > based), > > > > > > > > > both FP32 > > > > > > > > > > and INT8 precision on many topologies. > > > > > > > > > > > > > > > > > > > > -Ciyong > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > > > From: Zhao, Patric <patric.z...@intel.com> > > > > > > > > > > Sent: Tuesday, December 17, 2019 8:51 AM > > > > > > > > > > To: dev@mxnet.incubator.apache.org; d...@mxnet.apache.org > > > > > > > > > > Subject: RE: [VOTE] Release Apache MXNet (incubating) > > > > > > > > > > version > > > > > > > 1.6.0.rc0 > > > > > > > > > > Thanks, Tredak, I will add some words for the new feature in > > > > > > > > > > the > > > > > > > release > > > > > > > > > note. > > > > > > > > > > +1 for voting because we have ran multiple time of tests in > > > > > local and > > > > > > > > > got the > > > > > > > > > > expected performance boost. > > > > > > > > > > > > > > > > > > > > --Patric > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > > > > From: Przemysław Trędak <ptre...@apache.org> > > > > > > > > > > > Sent: Tuesday, December 17, 2019 4:49 AM > > > > > > > > > > > To: d...@mxnet.apache.org > > > > > > > > > > > Subject: [VOTE] Release Apache MXNet (incubating) version > > > > > 1.6.0.rc0 > > > > > > > > > > > Dear MXNet community, > > > > > > > > > > > > > > > > > > > > > > This is the vote to release Apache MXNet (incubating) > > > > > > > > > > > version > > > > > > > 1.6.0. > > > > > > > > > > > Voting starts now and will close on Friday, 20th December > > > > > > > > > > > 2019 > > > > > > > > > 23:59:59 PST. > > > > > > > > > > > Link to release notes: > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/MXNET/1.6.0+Release+notes > > > > > > > > > > > Link to release candidate: > > > > > > > > > > > > > > > > https://github.com/apache/incubator-mxnet/releases/tag/1.6.0.rc0 > > > > > > > > > > > Link to source and signatures on apache dist server: > > > > > > > > > > > > > > > > https://dist.apache.org/repos/dist/dev/incubator/mxnet/1.6.0.rc0/ > > > > > > > > > > > Please remember to TEST first before voting accordingly: > > > > > > > > > > > +1 = approve > > > > > > > > > > > +0 = no opinion > > > > > > > > > > > -1 = disapprove (provide reason) > > > > > > > > > > > > > > > > > > > > > > Additional notes: > > > > > > > > > > > - There was an issue[1] raised that 1.6.0.rc0 does not > > > > > > > > > > > build > > > > > with > > > > > > > > > > > clang on FreeBSD - I decided to not block the voting for > > > > > > > > > > > this > > > > > and > > > > > > > > > > > instead let the Community decide whether this is a blocker > > > > > > > > > > > for > > > > > the > > > > > > > > > release. > > > > > > > > > > > - Patric Zhao and Tao Lv - could you help preparing a > > > > > paragraph on > > > > > > > > > > > MKLDNN > > > > > > > > > > > 1.0 update in the New features section in the release > > > > > > > > > > > notes? > > > > > > > > > > > > > > > > > > > > > > [1] https://github.com/apache/incubator-mxnet/issues/17076 > > > > > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > Przemyslaw Tredak
