Re: [DISCUSS] improve MXNet Scala release process

2018-08-15 Thread Qing Lan
Hi all, I have created a design document on Automated Scala publish in here: https://cwiki.apache.org/confluence/display/MXNET/Automated+MXNet+Scala+release+design Please kindly review it and leave any thoughts you may have. Thanks, Qing On 8/3/18, 10:26 AM, "Naveen Swamy" wrote: Hi Cari

Re: [DISCUSS] improve MXNet Scala release process

2018-08-03 Thread Naveen Swamy
Hi Carin, The thinking right now is to publish nightly to the Apache Snapshot repository, building and validating with integration tests. We(Qing, me and Andrew Ayres) will work on a elaborate document detailing the process and we'll loop you in as well. Thanks, Naveen On Fri, Aug 3, 2018 at 8:3

Re: [DISCUSS] improve MXNet Scala release process

2018-08-03 Thread Carin Meier
Hi, I was thinking about the process for publishing the Clojure jar as well. I think, since it will be published to Nexus/Maven and the building of it depends on the Scala jar artifact, it might make sense to combine the publishing of the Clojure jar at the same time as the Scala Jar. I haven't wo

Re: [DISCUSS] improve MXNet Scala release process

2018-07-31 Thread Qing Lan
Upon offline discussion with Marco, He proposed a plan that can actually help us conduct 3): 1. This job will not be trigger when PR runs and strictly limit that only committer can run the restricted job. 2. The code being run in there will only covers the code from the branch yo

Re: [DISCUSS] improve MXNet Scala release process

2018-07-28 Thread Qing Lan
Thanks Marco, Naveen and Sheng's feedback. About the 1): Scala side will only pack the mxnet binary only and use dynamic links to all the rest dependencies. So indeed it will require users to install all deps as the same as the builder platforms version and this will make them hard to use. Let'

Re: [DISCUSS] improve MXNet Scala release process

2018-07-27 Thread Marco de Abreu
Hi all, about the credential management: We already have a solution based on restricted slaves [1] and AWS secrets manager [2] that is generally classified to generate binaries and handle credentials. It was designed with continuous deployment in mind, but we haven't tested it in that field yet.

Re: [DISCUSS] improve MXNet Scala release process

2018-07-27 Thread Sheng Zha
Thanks, Naveen. Once we have clarity on 3), it should be no problem for scala to reuse the same solution. For 1), if this is indeed an issue, it seems that we may have rushed a bit on the scala releases. Are there any user reports? -sz On Fri, Jul 27, 2018 at 5:26 PM, Naveen Swamy wrote: > I co

Re: [DISCUSS] improve MXNet Scala release process

2018-07-27 Thread Naveen Swamy
I collaborate with Qing as a part of my day time job, to give you a little more perspective on the proposed work For 1) What we found is that users often run into conflicts when they use a different version of the dependency(CUDA, CUDNN, OpenBLAS, OpenCV, etc,.) and the one we build with MXNet bac

Re: [DISCUSS] improve MXNet Scala release process

2018-07-27 Thread Sheng Zha
Qing, For 1, why would it be a blocker, given that there were previous releases? Has there been compatibility issues for scala packages? If so, why did we release? There are many maven packages that include binary already, so if we can find the binary for all dependency it's probably best to link