Hi all, I have created a design document on Automated Scala publish in here: https://cwiki.apache.org/confluence/display/MXNET/Automated+MXNet+Scala+release+design
Please kindly review it and leave any thoughts you may have. Thanks, Qing On 8/3/18, 10:26 AM, "Naveen Swamy" <mnnav...@gmail.com> wrote: Hi Carin, The thinking right now is to publish nightly to the Apache Snapshot repository, building and validating with integration tests. We(Qing, me and Andrew Ayres) will work on a elaborate document detailing the process and we'll loop you in as well. Thanks, Naveen On Fri, Aug 3, 2018 at 8:35 AM, Carin Meier <carinme...@gmail.com> wrote: > Hi, > > I was thinking about the process for publishing the Clojure jar as well. I > think, since it will be published to Nexus/Maven and the building of it > depends on the Scala jar artifact, it might make sense to combine the > publishing of the Clojure jar at the same time as the Scala Jar. I haven't > worked out all the details yet, but I'm thinking with that it would be a > couple command lines in the same script that handles the Scala deployment. > > Or if it is a separate process, it might be able to share common parts. > > Could you please keep me in the loop of the deployment process so we can > figure out the best place to work in Clojure as well? > > Thanks, > Carin > > On Tue, Jul 31, 2018 at 3:49 PM, Qing Lan <lanking...@live.com> wrote: > > > Upon offline discussion with Marco, > > > > He proposed a plan that can actually help us conduct 3): > > 1. This job will not be trigger when PR runs and strictly limit > > that only committer can run the restricted job. > > 2. The code being run in there will only covers the code from the > > branch you choose to go, it will be committers responsibilities not to > > merge any trivial credential grabber code. > > 3. Test this is simple. The restricted job uses a similar > > architecture with current CI. You can send a PR with dockerfiles, scripts > > and configurations on Jenkins to give it a test to run the job with a > mock > > credential. Finally please contact people working on CI to give it a test > > run and they will do the last step to merge your change to CI. > > 4. Marco also mentioned the security level of the credentials. > The > > credential being used in the AWS Credential services will be assigned > with > > an individual IAM role, which only allows to access to the credentials > that > > role being assigned to, and used in the restricted job you have set up. > > > > I would also like to encourage people in this list to join the > > https://cwiki.apache.org/confluence/display/MXNET/ > > MXNet+Berlin+Office+Hours as the people who is working on improving the > > CI are there ready to help. > > > > Thanks, > > Qing > > > > > > On 7/28/18, 11:44 PM, "Qing Lan" <lanking...@live.com> wrote: > > > > Thanks Marco, Naveen and Sheng's feedback. > > > > About the 1): Scala side will only pack the mxnet binary only and use > > dynamic links to all the rest dependencies. So indeed it will require > users > > to install all deps as the same as the builder platforms version and this > > will make them hard to use. Let's please collaborate and create a (set > of) > > general CI script(s) to install the deps and bring static links to the > > package. > > > > About 3): it is indeed a general problems for both Scala and Python > > publish. If there is a good way we can safely store the credentials, we > can > > definitely give automated publish a go. And thanks again for Marco's > option > > provided below, I think we can make use of the restricted slaves and give > > it a test run. And to Marco: > > 1. Will this restricted jobs being triggered in every PR runs or > > it just depends on where you put it (like I put in nightly it will never > > be trigger in PR)? Will there be a potential risk like a PR attack > > (create a PR to grab credentials) > > > > 2. How do we make sure the coding being run there is under > control > > and not be changed by anyone? > > > > 3. If I want to test this functionality, where is the best place > > to create the job and make a test run? > > > > Thanks, > > Qing > > > > > > > > On 7/27/18, 5:44 PM, "Marco de Abreu" <marco.g.ab...@googlemail.com. > INVALID> > > wrote: > > > > Hi all, > > > > about the credential management: We already have a solution based > > on > > restricted slaves [1] and AWS secrets manager [2] that is > generally > > classified to generate binaries and handle credentials. It was > > designed > > with continuous deployment in mind, but we haven't tested it in > > that field > > yet. > > > > To properly assess the requirements, it would be great if we have > > this > > security critical part outlined for each release pipeline. We > > could then > > check and see if our existing solution matches all requirements > or > > if > > further work is necessary. > > > > Best regards, > > Marco > > > > [1] > > https://cwiki.apache.org/confluence/display/MXNET/ > > Restricted+jobs+and+nodes > > [2] https://docs.aws.amazon.com/secretsmanager/latest/ > > userguide/intro.html > > > > On 7/27/18, 5:43 PM, "Sheng Zha" <szha....@gmail.com> wrote: > > > > Thanks, Naveen. Once we have clarity on 3), it should be no > > problem for > > scala to reuse the same solution. For 1), if this is indeed an > > issue, it > > seems that we may have rushed a bit on the scala releases. Are > > there any > > user reports? > > > > -sz > > > > On Fri, Jul 27, 2018 at 5:26 PM, Naveen Swamy < > mnnav...@gmail.com> > > wrote: > > > > > I collaborate with Qing as a part of my day time job, to give > > you a little > > > more perspective on the proposed work > > > > > > For 1) > > > What we found is that users often run into conflicts when they > > use a > > > different version of the dependency(CUDA, CUDNN, OpenBLAS, > > OpenCV, etc,.) > > > and the one we build with MXNet backend and use in the MXNet > > Scala package. > > > Also it makes its not very straight-forward for users to > install > > these > > > dependencies themselves in order to lower the entry barrier and > > to make > > > everything work out of the box we are thinking to build MXNet > > all these > > > dependencies with MXNet (as a static library) and embed them in > > the MXNet > > > Scala package. This is also inspired by the work you have done > > for Apache > > > MXNet pip packages, Ideally I would like to reuse some of that > > work. > > > > > > Maven does not manage the binaries, you still have to build the > > binary and > > > release them but what dependencies the binaries are built with > > is what > > > causes the confusion and problems. > > > In the past there were 20+ packages of MXNet and I reduced it > to > > 3 (OSX, > > > Linux-CPU, Linux-GPU -- please see the discussion thread > below), > > with the > > > latest version of the dependencies and we'll build/manage > > additional > > > packages based on the user demand. > > > > > > Please see the previous discussion about this topic. > > > > > > 1) Scala Maven Package discussion: > > > https://lists.apache.org/thread.html/ > > c3846515fc5560d826e7b6f47e9b8b > > > 6728a925e6f8decb9676456144@%3Cdev.mxnet.apache.org%3E > > > 2) Config for Scala Package: > > > https://lists.apache.org/thread.html/ > > 0be6beb89cc2a792e7ba861a251f9a > > > 9a0b81fa36a5a0cd59d9f2cb6f@%3Cdev.mxnet.apache.org%3E > > > 3) Current Scala Package Release process: > > > https://cwiki.apache.org/confluence/display/MXNET/ > > > MXNet-Scala+Release+Process > > > > > > > > > Hope that answers > > > > > > > > > On Fri, Jul 27, 2018 at 4:59 PM, Sheng Zha <szha....@gmail.com > > > > wrote: > > > > > > > Qing, > > > > > > > > For 1, why would it be a blocker, given that there were > > previous > > > releases? > > > > Has there been compatibility issues for scala packages? If > so, > > why did we > > > > release? > > > > There are many maven packages that include binary already, so > > if we can > > > > find the binary for all dependency it's probably best to link > > to them, > > > and > > > > let maven manage these dependencies. > > > > > > > > For 2, since the current CI is based on Jenkins, I imagine it > > would be > > > some > > > > sort of Jenkins pipeline? Other people who're better versed > in > > Jenkins > > > can > > > > chime in. > > > > > > > > Personally, I'm interested in 3 as well. I have a pipeline > for > > building > > > pip > > > > packages that's currently not utilizing the CI, and the item > 3 > > is the > > > > blocker too. Once you finish, it would be great to refer to > > the same > > > > solution, so that I can move it into the same CI. > > > > > > > > -sz > > > > > > > > On Fri, Jul 27, 2018 at 4:37 PM, Qing Lan < > lanking...@live.com> > > wrote: > > > > > > > > > Hi all, > > > > > > > > > > Recently contributors on Scala Language development worked > > together and > > > > > finally able to publish Scala package on Maven. Now I would > > like to > > > > raise a > > > > > discussion to automate Scala release process and also > > discover a > > > standard > > > > > way to release different packages for MXNet so we won’t ask > > any > > > > individuals > > > > > to spend a long time to publish the package. > > > > > > > > > > There are three blocks that stop this automated process: > > > > > > > > > > 1. How to build general hardware-compatible backend > > dependencies for > > > > > MXNet (Linux CPU/GPU Mac OSX) > > > > > 2. How to automate the frontend release process and CI > > integration > > > > > 3. How to keep credentials for the release in the > pipeline > > > > > > > > > > Scala Release process created by Naveen: > > https://cwiki.apache.org/ > > > > > confluence/display/MXNET/MXNet-Scala+Release+Process > > > > > > > > > > Thanks, > > > > > Qing > > > > > > > > > > > > > > > > > > > > > > > > > > > >