Thanks Pedro for you work to improve the CI setup. For Windows, let's see if fixing the deterministic failure of GPU tests with the updated toolchain helps to fix the non-deterministic failures in the old toolchain.
Best regards Leonard On Fri, 2020-02-21 at 15:37 -0800, Pedro Larroy wrote: > CI is back to normal. We haven't updated Windows AMIs due to issues with > GPU unit tests. > > You might need to retrigger your PRs. > > Thanks for your patience. > > On Wed, Feb 19, 2020 at 5:54 PM Pedro Larroy <pedro.larroy.li...@gmail.com> > wrote: > > > I reverted the CI rollout due to the following issues: > > > > https://github.com/apache/incubator-mxnet/issues/17633 > > > > https://github.com/apache/incubator-mxnet/issues/17635 > > > > I would need help from the community to fix them as we can't even compile > > in debug mode in windows as the above, and also due to older cmake being > > used in vs2017. > > > > For updating to vs2019 we would need to update cuda. > > > > Pedro. > > > > > > > > On Tue, Feb 18, 2020 at 5:31 PM Pedro Larroy <pedro.larroy.li...@gmail.com> > > wrote: > > > > > Hi > > > > > > Tomorrow I will be updating the CI environment with new AMIs, and > > > deploying updated autoscaling logic with fixes, expect some disruptions in > > > CI runs. > > > > > > The Linux AMIs will be updated to Ubuntu 18.04 with updated GPU drivers, > > > this won't affect Linux container builds. > > > > > > The new Windows AMI comes with a reproducible environment, VS2017, Visual > > > C++ updated from VC14 to VC15. > > > > > > CMake 3.16.2, Perl and LLVM which are required for MXNet and TVM. Cuda is > > > still 9.2, but now it's easier to update as the installation is automated. > > > > > > Once the environment is updated, my PR needs to be merged to bring back > > > windows compilation in working order: > > > > > > https://github.com/apache/incubator-mxnet/pull/17206 > > > > > > Thanks to Leonard and Joe for helping with various issues. > > > > > > Pedro. > > >
