Looks good! +1 On Tue, Jan 9, 2018 at 10:24 AM, kellen sunderland < kellen.sunderl...@gmail.com> wrote:
> I think most were in favour of at a minimum creating a clang build so I've > created a PR > https://github.com/apache/incubator-mxnet/pull/9330/commits/ > 84089ea14123ebe4d66cc92e82a2d529cfbd8b19. > My hope is this will catch many of the issues blocking OSX builds. In fact > it already caught one issue. If you guys are in favour I can remove the > WIP and ask that it be merged. > > On Thu, Jan 4, 2018 at 6:29 PM, Chris Olivier <cjolivie...@gmail.com> > wrote: > > > Nope, I have been on vacation. > > > > On Thu, Jan 4, 2018 at 9:10 AM, kellen sunderland < > > kellen.sunderl...@gmail.com> wrote: > > > > > Hope everyone had a good break. Just wanted to check if there were > > further > > > thoughts on OSX builds. Chris, did you have time to look into > > virtualizing > > > Mac OS? Would it make sense for us to put something in place in the > > > interim e.g. the clang solution? > > > > > > On Tue, Dec 12, 2017 at 7:59 PM, de Abreu, Marco <mab...@amazon.com> > > > wrote: > > > > > > > Thanks for looking into this, Chris! No hurries on that one, we’ll > look > > > > into it next stage when we add new system- and build-configurations > to > > > the > > > > CI. > > > > > > > > On 12.12.17, 19:12, "Chris Olivier" <cjolivie...@gmail.com> wrote: > > > > > > > > I am on vacation starting Thursday. > > > > > > > > On Tue, Dec 12, 2017 at 9:49 AM kellen sunderland < > > > > kellen.sunderl...@gmail.com> wrote: > > > > > > > > > Absolutely, let's do an investigation and see if it's possible > to > > > > > virtualize. Would you have time to look into it a bit further? > > > > > > > > > > On Tue, Dec 12, 2017 at 6:47 PM, Chris Olivier < > > > > cjolivie...@gmail.com> > > > > > wrote: > > > > > > > > > > > Don’t get me wrong, I’m not saying this Mac OS Jenkins > solution > > > is > > > > doable > > > > > > but I feel like we should investigate because the payoff > would > > be > > > > large. > > > > > > > > > > > > > > > > > > On Tue, Dec 12, 2017 at 9:38 AM Chris Olivier < > > > > cjolivie...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > Apple’s Darwin OS Is recently open-sourced. > > > > > > > https://github.com/PureDarwin/PureDarwin > > > > > > > > > > > > > > How to convert this into a non-GUI VM I am not sure but I > am > > > > willing to > > > > > > > bet that people have done it already. > > > > > > > > > > > > > > On Tue, Dec 12, 2017 at 9:16 AM kellen sunderland < > > > > > > > kellen.sunderl...@gmail.com> wrote: > > > > > > > > > > > > > >> It might be technically possible, but I think it would > > violate > > > > the > > > > > MacOS > > > > > > >> license: http://store.apple.com/ > > Catalog/US/Images/MacOSX.htm > > > > > > >> > > > > > > >> "2. Permitted License Uses and Restrictions. > > > > > > >> A. This License allows you to install and use one copy of > > the > > > > Apple > > > > > > >> Software on a single Apple-labeled computer at a time. > This > > > > License > > > > > does > > > > > > >> not allow the Apple Software to exist on more than one > > > computer > > > > at a > > > > > > >> time,and you may not make the Apple Software available > over > > a > > > > network > > > > > > >> where > > > > > > >> it could be used by multiple computers at the same time. > You > > > > may make > > > > > > one > > > > > > >> copy of the Apple Software (excluding the Boot ROM code) > in > > > > > > >> machine-readable form for backup purposes only; provided > > that > > > > the > > > > > backup > > > > > > >> copy must include all copyright or other proprietary > notices > > > > contained > > > > > > on > > > > > > >> the original. " > > > > > > >> > > > > > > >> I could be wrong though, does anyone know the details of > > MacOS > > > > > > licensing / > > > > > > >> virtualization? > > > > > > >> > > > > > > >> On Tue, Dec 12, 2017 at 6:10 PM, Chris Olivier < > > > > cjolivie...@gmail.com > > > > > > > > > > > > >> wrote: > > > > > > >> > > > > > > >> > googling seems to be full of running OSX (and even > > > > open-sourced > > > > > > >> PureDarwin) > > > > > > >> > in VMs. One could conceivably run a VM on an EC2 > instance, > > > > right? > > > > > > >> > > > > > > > >> > On Tue, Dec 12, 2017 at 9:01 AM kellen sunderland < > > > > > > >> > kellen.sunderl...@gmail.com> wrote: > > > > > > >> > > > > > > > >> > > It would be ideal if we could cover OSX in Jenkins, > but > > > the > > > > only > > > > > > >> solution > > > > > > >> > > that I'm aware of would require physical machines to > be > > > the > > > > > workers. > > > > > > >> I > > > > > > >> > > would be weakly opposed to having physical servers > > running > > > > on PRs. > > > > > > >> The > > > > > > >> > > downsides that I see in order of importance: > > > > > > >> > > > > > > > > >> > > - We can't autoscale physical hardware. If we find > > that > > > > the > > > > > load > > > > > > is > > > > > > >> > too > > > > > > >> > > high we have to buy more machines. > > > > > > >> > > - Security would be tricky, as they'd have to be > > > connected > > > > to the > > > > > > >> > internet > > > > > > >> > > and then to our Jekins master instance. Connecting > via > > a > > > > wired > > > > > > >> network > > > > > > >> > > would probably not be possible on most corporate > > networks > > > > as these > > > > > > >> > machines > > > > > > >> > > are by definition running arbitrary code from the > > > > internet. Many > > > > > > >> > corporate > > > > > > >> > > sites have public wifi that this machine could > > potentially > > > > connect > > > > > > to, > > > > > > >> > but > > > > > > >> > > then our PRs start failing if the wifi disconnects > > > > temporarily. > > > > > To > > > > > > >> > connect > > > > > > >> > > to the master we would need to setup a vpn solution > with > > > > endpoints > > > > > > in > > > > > > >> our > > > > > > >> > > vpc on AWS. This is possible but would probably > > require a > > > > lot of > > > > > > >> > security > > > > > > >> > > work. > > > > > > >> > > - We can't just create a simple startup script or > yaml > > > > file that > > > > > is > > > > > > >> > > checked into GitHub to manage the machine. Someone > will > > > > actually > > > > > > >> have to > > > > > > >> > > physically administer the machine, apply updates, etc. > > > > which will > > > > > > make > > > > > > >> > > community ownership difficult. > > > > > > >> > > > > > > > > >> > > Specific to an OSX build: > > > > > > >> > > - We can't virtualize OSX which means we'd only be > able > > > to > > > > cover > > > > > > one > > > > > > >> OSX > > > > > > >> > > build environment per physical device. We couldn't > > > target a > > > > > matrix > > > > > > of > > > > > > >> > OSX > > > > > > >> > > and Xcode versions as in Travis. > > > > > > >> > > > > > > > > >> > > -Kellen > > > > > > >> > > > > > > > > >> > > On Tue, Dec 12, 2017 at 5:46 PM, Chris Olivier < > > > > > > cjolivie...@gmail.com > > > > > > >> > > > > > > > >> > > wrote: > > > > > > >> > > > > > > > > >> > > > So why Travis when we could possibly use Jenkins? > > > > > > >> > > > > > > > > > >> > > > On Tue, Dec 12, 2017 at 7:59 AM Marco de Abreu < > > > > > > >> > > > marco.g.ab...@googlemail.com> > > > > > > >> > > > wrote: > > > > > > >> > > > > > > > > > >> > > > > Yes that's correct, Chris. > > > > > > >> > > > > > > > > > > >> > > > > Am 12.12.2017 4:46 nachm. schrieb "Chris Olivier" > < > > > > > > >> > > cjolivie...@gmail.com > > > > > > >> > > > >: > > > > > > >> > > > > > > > > > > >> > > > > > A quick google search seems to indicate that Mac > > can > > > > be used > > > > > > as > > > > > > >> a > > > > > > >> > > > Jenkins > > > > > > >> > > > > > slave. Is this correct? > > > > > > >> > > > > > > > > > > > >> > > > > > On Tue, Dec 12, 2017 at 7:42 AM Steffen Rochel < > > > > > > >> > > > steffenroc...@gmail.com> > > > > > > >> > > > > > wrote: > > > > > > >> > > > > > > > > > > > >> > > > > > > +1 for #1 and #2 > > > > > > >> > > > > > > > > > > > > >> > > > > > > I’m working on getting a MacPro to add to CI > > > system. > > > > > > >> > > > > > > On Tue, Dec 12, 2017 at 1:43 AM kellen > > sunderland > > > < > > > > > > >> > > > > > > kellen.sunderl...@gmail.com> wrote: > > > > > > >> > > > > > > > > > > > > >> > > > > > > > Background: TravisCI is a startup providing > > > > managed > > > > > > >> continuous > > > > > > >> > > > > > > > integration services with GitHub integration > > and > > > > YAML > > > > > > based > > > > > > >> > > > > > > configuration. > > > > > > >> > > > > > > > TravisCI is one of the few CI providers that > > > will > > > > build > > > > > a > > > > > > >> > variety > > > > > > >> > > > of > > > > > > >> > > > > > > > OSX/MacOS builds for software projects. > Their > > > > pricing > > > > > > >> ranges > > > > > > >> > > from > > > > > > >> > > > > Free > > > > > > >> > > > > > > > (for open source, 1 concurrent job, to $489 > > > > monthly for > > > > > 10 > > > > > > >> > > > concurrent > > > > > > >> > > > > > > jobs). > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > Problem: We’ve had a few OSX build issues > slip > > > > into > > > > > MXNet > > > > > > >> > master > > > > > > >> > > in > > > > > > >> > > > > the > > > > > > >> > > > > > > > past few weeks. We’ve previously had a > Travis > > > CI > > > > based > > > > > > >> testing > > > > > > >> > > > > system > > > > > > >> > > > > > > that > > > > > > >> > > > > > > > would have caught these issues. > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > Proposals so far: > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > 1) Use TravisCI in it’s free mode for a very > > > > minimal > > > > > > sanity > > > > > > >> > check > > > > > > >> > > > on > > > > > > >> > > > > > OSX. > > > > > > >> > > > > > > > If we compile the program, and for example > run > > > > C++ unit > > > > > > >> tests > > > > > > >> > > we’re > > > > > > >> > > > > > > > unlikely to run into problems with queued > > > > builds. The > > > > > > total > > > > > > >> > > build > > > > > > >> > > > > time > > > > > > >> > > > > > > > here should be less than 15 minutes. > > > > Configuration > > > > > should > > > > > > >> be > > > > > > >> > > quite > > > > > > >> > > > > > > simple > > > > > > >> > > > > > > > and easy to maintain. Error messages should > > > also > > > > be > > > > > > >> obvious to > > > > > > >> > > > > > > > contributors. > > > > > > >> > > > > > > > 2) Run clang in Linux with our current CI. > > > > Building > > > > > with > > > > > > >> clang > > > > > > >> > > > > should > > > > > > >> > > > > > > > take less than 10 minutes, should flush out > a > > > > large > > > > > subset > > > > > > >> of > > > > > > >> > the > > > > > > >> > > > > > issues > > > > > > >> > > > > > > > we’ve seen with OSX, and be quite easy to > > > > maintain. > > > > > > >> > > > > > > > 3) Run full test-suites in TravisCI, > equaling > > > the > > > > level > > > > > of > > > > > > >> > > coverage > > > > > > >> > > > > we > > > > > > >> > > > > > > > provide to Linux in Jenkins. This could > > require > > > > us to > > > > > > >> > subscribe > > > > > > >> > > > to a > > > > > > >> > > > > > > > monthly package with Travis to ensure our > > build > > > > queue > > > > > > >> doesn’t > > > > > > >> > > grow > > > > > > >> > > > to > > > > > > >> > > > > > an > > > > > > >> > > > > > > > unacceptable length. It may also require a > > > > volunteer to > > > > > > >> setup > > > > > > >> > > and > > > > > > >> > > > > > > maintain > > > > > > >> > > > > > > > long-term. > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > I’d +1 #1 and #2 as I think those should be > > > > low-cost, > > > > > > >> > > low-maintence > > > > > > >> > > > > > > > solutions that should catch the majority of > > the > > > > problems > > > > > > >> we’ve > > > > > > >> > > seen > > > > > > >> > > > > > thus > > > > > > >> > > > > > > > far. > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > -Kellen > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
