Re: Release blocker: non-determinstic forward in gluon

I'd agree that we should have a repeatable process for generating artifacts. It would be useful for Apache release reviewers to be able to double check the results we get in CI, and it would help give a consistent experience for users. I'm a little uncomfortable with the idea of generating the ac

Re: Release blocker: non-determinstic forward in gluon

Thanks Pedro. Good to know you think it is important as well. I hope the community can review a proposal on the CWiki soon? that would be great... On Mon, Jul 30, 2018 at 4:26 AM Pedro Larroy wrote: > Hi Hagay > > We are aware of this and we are working in this direction which as you > point out

Re: Release blocker: non-determinstic forward in gluon

Hi Hagay We are aware of this and we are working in this direction which as you point out, is more desirable. There's a huge amount of non-trivial work that has gone into building these distribution packages from Sheng which needs to be adapted for our CI system, and taken into consideration. Ped

Re: Release blocker: non-determinstic forward in gluon

Thanks Tong for root-causing the issue! Thanks Sheng for following up with an updated PyPi package. What worries me is that we seem to build MXNet PyPi distribution packages with a build config different than the CI where all of the tests are running. Looking here [1

Re: Release blocker: non-determinstic forward in gluon

Tong, That's great news. I'm glad that OpenBLAS people are responding so quickly. In that case it's probably a better idea to use that version instead. The latest OpenBLAS version brings many optimization for all kinds of hardware. -sz On Fri, Jul 27, 2018 at 11:10 AM, Tong He wrote: > Hi Shen

Re: Release blocker: non-determinstic forward in gluon

Hi Sheng, I also opened an issue on OpenBLAS repo: https://github.com/xianyi/OpenBLAS/issues/1700 . As informed that "0.3.2 should be released this weekend", I tested their develope branch as well, and seems the new version has fixed the bug. Since OpenBLAS 0.3.2 could also have performance impr

Re: Release blocker: non-determinstic forward in gluon

Forgot to mention, the post release version is a pip package version. -sz > On Jul 27, 2018, at 10:42 AM, Sheng Zha wrote: > > In this case we can regard it as a release problem, which is usually what > post release versions are for. It’s still the same release with different > dependency, so

Re: Release blocker: non-determinstic forward in gluon

In this case we can regard it as a release problem, which is usually what post release versions are for. It’s still the same release with different dependency, so there is no code change needed. -sz > On Jul 27, 2018, at 8:31 AM, Steffen Rochel wrote: > > Hi Tong - thanks for root causing th

Re: Release blocker: non-determinstic forward in gluon

Hi Tong - thanks for root causing the problem. Sheng - what is 1.2.1.post0? Shouldn't a patch with fix be released as 1.2.2? Steffen On Thu, Jul 26, 2018 at 5:33 PM Sheng Zha wrote: > Dear users and developers of Apache MXNet (Incubating), > > Thanks to Tong's dedication, the root cause for this

Re: Release blocker: non-determinstic forward in gluon

Dear users and developers of Apache MXNet (Incubating), Thanks to Tong's dedication, the root cause for this issue was identified to be instability in OpenBLAS's latest stable version 0.3.1. For details, see Tong's comment