Re: [DISCUSS] improve MXNet Scala release process
Hi all, I have created a design document on Automated Scala publish in here: https://cwiki.apache.org/confluence/display/MXNET/Automated+MXNet+Scala+release+design Please kindly review it and leave any thoughts you may have. Thanks, Qing On 8/3/18, 10:26 AM, "Naveen Swamy" wrote: Hi Carin, The thinking right now is to publish nightly to the Apache Snapshot repository, building and validating with integration tests. We(Qing, me and Andrew Ayres) will work on a elaborate document detailing the process and we'll loop you in as well. Thanks, Naveen On Fri, Aug 3, 2018 at 8:35 AM, Carin Meier wrote: > Hi, > > I was thinking about the process for publishing the Clojure jar as well. I > think, since it will be published to Nexus/Maven and the building of it > depends on the Scala jar artifact, it might make sense to combine the > publishing of the Clojure jar at the same time as the Scala Jar. I haven't > worked out all the details yet, but I'm thinking with that it would be a > couple command lines in the same script that handles the Scala deployment. > > Or if it is a separate process, it might be able to share common parts. > > Could you please keep me in the loop of the deployment process so we can > figure out the best place to work in Clojure as well? > > Thanks, > Carin > > On Tue, Jul 31, 2018 at 3:49 PM, Qing Lan wrote: > > > Upon offline discussion with Marco, > > > > He proposed a plan that can actually help us conduct 3): > > 1. This job will not be trigger when PR runs and strictly limit > > that only committer can run the restricted job. > > 2. The code being run in there will only covers the code from the > > branch you choose to go, it will be committers responsibilities not to > > merge any trivial credential grabber code. > > 3. Test this is simple. The restricted job uses a similar > > architecture with current CI. You can send a PR with dockerfiles, scripts > > and configurations on Jenkins to give it a test to run the job with a > mock > > credential. Finally please contact people working on CI to give it a test > > run and they will do the last step to merge your change to CI. > > 4. Marco also mentioned the security level of the credentials. > The > > credential being used in the AWS Credential services will be assigned > with > > an individual IAM role, which only allows to access to the credentials > that > > role being assigned to, and used in the restricted job you have set up. > > > > I would also like to encourage people in this list to join the > > https://cwiki.apache.org/confluence/display/MXNET/ > > MXNet+Berlin+Office+Hours as the people who is working on improving the > > CI are there ready to help. > > > > Thanks, > > Qing > > > > > > On 7/28/18, 11:44 PM, "Qing Lan" wrote: > > > > Thanks Marco, Naveen and Sheng's feedback. > > > > About the 1): Scala side will only pack the mxnet binary only and use > > dynamic links to all the rest dependencies. So indeed it will require > users > > to install all deps as the same as the builder platforms version and this > > will make them hard to use. Let's please collaborate and create a (set > of) > > general CI script(s) to install the deps and bring static links to the > > package. > > > > About 3): it is indeed a general problems for both Scala and Python > > publish. If there is a good way we can safely store the credentials, we > can > > definitely give automated publish a go. And thanks again for Marco's > option > > provided below, I think we can make use of the restricted slaves and give > > it a test run. And to Marco: > > 1. Will this restricted jobs being triggered in every PR runs or > > it just depends on where you put it (like I put in nightly it will never > >be trigger in PR)? Will there be a potential risk like a PR attack > > (create a PR to grab credentials) > > > > 2. How do we make sure the coding being run there is under > control > > and not be changed by anyone? > > > > 3. If I want to test this functionality, where is the best place > > to create the job and make a test run? > > > > Thanks, > > Qing > > > > > > > > On 7/27/18, 5:44 PM, "Marco de Abreu" INVALID> > > wrote: > > > > Hi all, > > > > about the credential management: We already have a solution based > > on > > restricted slaves [1] and AWS secrets manager [2] that is > generally > > classified to generate binaries and handle credentials. It was > > designed > > with continuous
Re: [DISCUSS] improve MXNet Scala release process
Hi Carin, The thinking right now is to publish nightly to the Apache Snapshot repository, building and validating with integration tests. We(Qing, me and Andrew Ayres) will work on a elaborate document detailing the process and we'll loop you in as well. Thanks, Naveen On Fri, Aug 3, 2018 at 8:35 AM, Carin Meier wrote: > Hi, > > I was thinking about the process for publishing the Clojure jar as well. I > think, since it will be published to Nexus/Maven and the building of it > depends on the Scala jar artifact, it might make sense to combine the > publishing of the Clojure jar at the same time as the Scala Jar. I haven't > worked out all the details yet, but I'm thinking with that it would be a > couple command lines in the same script that handles the Scala deployment. > > Or if it is a separate process, it might be able to share common parts. > > Could you please keep me in the loop of the deployment process so we can > figure out the best place to work in Clojure as well? > > Thanks, > Carin > > On Tue, Jul 31, 2018 at 3:49 PM, Qing Lan wrote: > > > Upon offline discussion with Marco, > > > > He proposed a plan that can actually help us conduct 3): > > 1. This job will not be trigger when PR runs and strictly limit > > that only committer can run the restricted job. > > 2. The code being run in there will only covers the code from the > > branch you choose to go, it will be committers responsibilities not to > > merge any trivial credential grabber code. > > 3. Test this is simple. The restricted job uses a similar > > architecture with current CI. You can send a PR with dockerfiles, scripts > > and configurations on Jenkins to give it a test to run the job with a > mock > > credential. Finally please contact people working on CI to give it a test > > run and they will do the last step to merge your change to CI. > > 4. Marco also mentioned the security level of the credentials. > The > > credential being used in the AWS Credential services will be assigned > with > > an individual IAM role, which only allows to access to the credentials > that > > role being assigned to, and used in the restricted job you have set up. > > > > I would also like to encourage people in this list to join the > > https://cwiki.apache.org/confluence/display/MXNET/ > > MXNet+Berlin+Office+Hours as the people who is working on improving the > > CI are there ready to help. > > > > Thanks, > > Qing > > > > > > On 7/28/18, 11:44 PM, "Qing Lan" wrote: > > > > Thanks Marco, Naveen and Sheng's feedback. > > > > About the 1): Scala side will only pack the mxnet binary only and use > > dynamic links to all the rest dependencies. So indeed it will require > users > > to install all deps as the same as the builder platforms version and this > > will make them hard to use. Let's please collaborate and create a (set > of) > > general CI script(s) to install the deps and bring static links to the > > package. > > > > About 3): it is indeed a general problems for both Scala and Python > > publish. If there is a good way we can safely store the credentials, we > can > > definitely give automated publish a go. And thanks again for Marco's > option > > provided below, I think we can make use of the restricted slaves and give > > it a test run. And to Marco: > > 1. Will this restricted jobs being triggered in every PR runs or > > it just depends on where you put it (like I put in nightly it will never > >be trigger in PR)? Will there be a potential risk like a PR attack > > (create a PR to grab credentials) > > > > 2. How do we make sure the coding being run there is under > control > > and not be changed by anyone? > > > > 3. If I want to test this functionality, where is the best place > > to create the job and make a test run? > > > > Thanks, > > Qing > > > > > > > > On 7/27/18, 5:44 PM, "Marco de Abreu" INVALID> > > wrote: > > > > Hi all, > > > > about the credential management: We already have a solution based > > on > > restricted slaves [1] and AWS secrets manager [2] that is > generally > > classified to generate binaries and handle credentials. It was > > designed > > with continuous deployment in mind, but we haven't tested it in > > that field > > yet. > > > > To properly assess the requirements, it would be great if we have > > this > > security critical part outlined for each release pipeline. We > > could then > > check and see if our existing solution matches all requirements > or > > if > > further work is necessary. > > > > Best regards, > > Marco > > > > [1] > > https://cwiki.apache.org/confluence/display/MXNET/ > > Restricted+jobs+and+nodes > > [2] https://docs.aws.amazon.com/secretsmanager/latest/ > > userguide/intro.html > > > > On 7/27/18, 5:43 PM, "Sheng Zha" wrote: > > > > Thanks, Naveen. Once we
Re: [DISCUSS] improve MXNet Scala release process
Hi, I was thinking about the process for publishing the Clojure jar as well. I think, since it will be published to Nexus/Maven and the building of it depends on the Scala jar artifact, it might make sense to combine the publishing of the Clojure jar at the same time as the Scala Jar. I haven't worked out all the details yet, but I'm thinking with that it would be a couple command lines in the same script that handles the Scala deployment. Or if it is a separate process, it might be able to share common parts. Could you please keep me in the loop of the deployment process so we can figure out the best place to work in Clojure as well? Thanks, Carin On Tue, Jul 31, 2018 at 3:49 PM, Qing Lan wrote: > Upon offline discussion with Marco, > > He proposed a plan that can actually help us conduct 3): > 1. This job will not be trigger when PR runs and strictly limit > that only committer can run the restricted job. > 2. The code being run in there will only covers the code from the > branch you choose to go, it will be committers responsibilities not to > merge any trivial credential grabber code. > 3. Test this is simple. The restricted job uses a similar > architecture with current CI. You can send a PR with dockerfiles, scripts > and configurations on Jenkins to give it a test to run the job with a mock > credential. Finally please contact people working on CI to give it a test > run and they will do the last step to merge your change to CI. > 4. Marco also mentioned the security level of the credentials. The > credential being used in the AWS Credential services will be assigned with > an individual IAM role, which only allows to access to the credentials that > role being assigned to, and used in the restricted job you have set up. > > I would also like to encourage people in this list to join the > https://cwiki.apache.org/confluence/display/MXNET/ > MXNet+Berlin+Office+Hours as the people who is working on improving the > CI are there ready to help. > > Thanks, > Qing > > > On 7/28/18, 11:44 PM, "Qing Lan" wrote: > > Thanks Marco, Naveen and Sheng's feedback. > > About the 1): Scala side will only pack the mxnet binary only and use > dynamic links to all the rest dependencies. So indeed it will require users > to install all deps as the same as the builder platforms version and this > will make them hard to use. Let's please collaborate and create a (set of) > general CI script(s) to install the deps and bring static links to the > package. > > About 3): it is indeed a general problems for both Scala and Python > publish. If there is a good way we can safely store the credentials, we can > definitely give automated publish a go. And thanks again for Marco's option > provided below, I think we can make use of the restricted slaves and give > it a test run. And to Marco: > 1. Will this restricted jobs being triggered in every PR runs or > it just depends on where you put it (like I put in nightly it will never >be trigger in PR)? Will there be a potential risk like a PR attack > (create a PR to grab credentials) > > 2. How do we make sure the coding being run there is under control > and not be changed by anyone? > > 3. If I want to test this functionality, where is the best place > to create the job and make a test run? > > Thanks, > Qing > > > > On 7/27/18, 5:44 PM, "Marco de Abreu" > > wrote: > > Hi all, > > about the credential management: We already have a solution based > on > restricted slaves [1] and AWS secrets manager [2] that is generally > classified to generate binaries and handle credentials. It was > designed > with continuous deployment in mind, but we haven't tested it in > that field > yet. > > To properly assess the requirements, it would be great if we have > this > security critical part outlined for each release pipeline. We > could then > check and see if our existing solution matches all requirements or > if > further work is necessary. > > Best regards, > Marco > > [1] > https://cwiki.apache.org/confluence/display/MXNET/ > Restricted+jobs+and+nodes > [2] https://docs.aws.amazon.com/secretsmanager/latest/ > userguide/intro.html > > On 7/27/18, 5:43 PM, "Sheng Zha" wrote: > > Thanks, Naveen. Once we have clarity on 3), it should be no > problem for > scala to reuse the same solution. For 1), if this is indeed an > issue, it > seems that we may have rushed a bit on the scala releases. Are > there any > user reports? > > -sz > > On Fri, Jul 27, 2018 at 5:26 PM, Naveen Swamy > wrote: > > > I collaborate with Qing as a part of my day time job, to give > you a little > > more perspective on the proposed work > > > > For 1) > > What we found is that users often run into conflicts when
Re: [DISCUSS] improve MXNet Scala release process
Upon offline discussion with Marco, He proposed a plan that can actually help us conduct 3): 1. This job will not be trigger when PR runs and strictly limit that only committer can run the restricted job. 2. The code being run in there will only covers the code from the branch you choose to go, it will be committers responsibilities not to merge any trivial credential grabber code. 3. Test this is simple. The restricted job uses a similar architecture with current CI. You can send a PR with dockerfiles, scripts and configurations on Jenkins to give it a test to run the job with a mock credential. Finally please contact people working on CI to give it a test run and they will do the last step to merge your change to CI. 4. Marco also mentioned the security level of the credentials. The credential being used in the AWS Credential services will be assigned with an individual IAM role, which only allows to access to the credentials that role being assigned to, and used in the restricted job you have set up. I would also like to encourage people in this list to join the https://cwiki.apache.org/confluence/display/MXNET/MXNet+Berlin+Office+Hours as the people who is working on improving the CI are there ready to help. Thanks, Qing On 7/28/18, 11:44 PM, "Qing Lan" wrote: Thanks Marco, Naveen and Sheng's feedback. About the 1): Scala side will only pack the mxnet binary only and use dynamic links to all the rest dependencies. So indeed it will require users to install all deps as the same as the builder platforms version and this will make them hard to use. Let's please collaborate and create a (set of) general CI script(s) to install the deps and bring static links to the package. About 3): it is indeed a general problems for both Scala and Python publish. If there is a good way we can safely store the credentials, we can definitely give automated publish a go. And thanks again for Marco's option provided below, I think we can make use of the restricted slaves and give it a test run. And to Marco: 1. Will this restricted jobs being triggered in every PR runs or it just depends on where you put it (like I put in nightly it will never be trigger in PR)? Will there be a potential risk like a PR attack (create a PR to grab credentials) 2. How do we make sure the coding being run there is under control and not be changed by anyone? 3. If I want to test this functionality, where is the best place to create the job and make a test run? Thanks, Qing On 7/27/18, 5:44 PM, "Marco de Abreu" wrote: Hi all, about the credential management: We already have a solution based on restricted slaves [1] and AWS secrets manager [2] that is generally classified to generate binaries and handle credentials. It was designed with continuous deployment in mind, but we haven't tested it in that field yet. To properly assess the requirements, it would be great if we have this security critical part outlined for each release pipeline. We could then check and see if our existing solution matches all requirements or if further work is necessary. Best regards, Marco [1] https://cwiki.apache.org/confluence/display/MXNET/Restricted+jobs+and+nodes [2] https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html On 7/27/18, 5:43 PM, "Sheng Zha" wrote: Thanks, Naveen. Once we have clarity on 3), it should be no problem for scala to reuse the same solution. For 1), if this is indeed an issue, it seems that we may have rushed a bit on the scala releases. Are there any user reports? -sz On Fri, Jul 27, 2018 at 5:26 PM, Naveen Swamy wrote: > I collaborate with Qing as a part of my day time job, to give you a little > more perspective on the proposed work > > For 1) > What we found is that users often run into conflicts when they use a > different version of the dependency(CUDA, CUDNN, OpenBLAS, OpenCV, etc,.) > and the one we build with MXNet backend and use in the MXNet Scala package. > Also it makes its not very straight-forward for users to install these > dependencies themselves in order to lower the entry barrier and to make > everything work out of the box we are thinking to build MXNet all these > dependencies with MXNet (as a static library) and embed them in the MXNet > Scala package. This is also inspired by the work you have done for Apache > MXNet pip packages, Ideally I would like to reuse some of that work. > > Maven does not manage the binaries, you still have to build the binary and
Re: [DISCUSS] improve MXNet Scala release process
Thanks Marco, Naveen and Sheng's feedback. About the 1): Scala side will only pack the mxnet binary only and use dynamic links to all the rest dependencies. So indeed it will require users to install all deps as the same as the builder platforms version and this will make them hard to use. Let's please collaborate and create a (set of) general CI script(s) to install the deps and bring static links to the package. About 3): it is indeed a general problems for both Scala and Python publish. If there is a good way we can safely store the credentials, we can definitely give automated publish a go. And thanks again for Marco's option provided below, I think we can make use of the restricted slaves and give it a test run. And to Marco: 1. Will this restricted jobs being triggered in every PR runs or it just depends on where you put it (like I put in nightly it will never be trigger in PR)? Will there be a potential risk like a PR attack (create a PR to grab credentials) 2. How do we make sure the coding being run there is under control and not be changed by anyone? 3. If I want to test this functionality, where is the best place to create the job and make a test run? Thanks, Qing On 7/27/18, 5:44 PM, "Marco de Abreu" wrote: Hi all, about the credential management: We already have a solution based on restricted slaves [1] and AWS secrets manager [2] that is generally classified to generate binaries and handle credentials. It was designed with continuous deployment in mind, but we haven't tested it in that field yet. To properly assess the requirements, it would be great if we have this security critical part outlined for each release pipeline. We could then check and see if our existing solution matches all requirements or if further work is necessary. Best regards, Marco [1] https://cwiki.apache.org/confluence/display/MXNET/Restricted+jobs+and+nodes [2] https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html On 7/27/18, 5:43 PM, "Sheng Zha" wrote: Thanks, Naveen. Once we have clarity on 3), it should be no problem for scala to reuse the same solution. For 1), if this is indeed an issue, it seems that we may have rushed a bit on the scala releases. Are there any user reports? -sz On Fri, Jul 27, 2018 at 5:26 PM, Naveen Swamy wrote: > I collaborate with Qing as a part of my day time job, to give you a little > more perspective on the proposed work > > For 1) > What we found is that users often run into conflicts when they use a > different version of the dependency(CUDA, CUDNN, OpenBLAS, OpenCV, etc,.) > and the one we build with MXNet backend and use in the MXNet Scala package. > Also it makes its not very straight-forward for users to install these > dependencies themselves in order to lower the entry barrier and to make > everything work out of the box we are thinking to build MXNet all these > dependencies with MXNet (as a static library) and embed them in the MXNet > Scala package. This is also inspired by the work you have done for Apache > MXNet pip packages, Ideally I would like to reuse some of that work. > > Maven does not manage the binaries, you still have to build the binary and > release them but what dependencies the binaries are built with is what > causes the confusion and problems. > In the past there were 20+ packages of MXNet and I reduced it to 3 (OSX, > Linux-CPU, Linux-GPU -- please see the discussion thread below), with the > latest version of the dependencies and we'll build/manage additional > packages based on the user demand. > > Please see the previous discussion about this topic. > > 1) Scala Maven Package discussion: > https://lists.apache.org/thread.html/c3846515fc5560d826e7b6f47e9b8b > 6728a925e6f8decb9676456144@%3Cdev.mxnet.apache.org%3E > 2) Config for Scala Package: > https://lists.apache.org/thread.html/0be6beb89cc2a792e7ba861a251f9a > 9a0b81fa36a5a0cd59d9f2cb6f@%3Cdev.mxnet.apache.org%3E > 3) Current Scala Package Release process: > https://cwiki.apache.org/confluence/display/MXNET/ > MXNet-Scala+Release+Process > > > Hope that answers > > > On Fri, Jul 27, 2018 at 4:59 PM, Sheng Zha wrote: > > > Qing, > > > > For 1, why would it be a blocker, given that there were previous > releases? > > Has there been compatibility issues for scala packages? If so, why did we > > release? > > There are many maven packages that include binary already, so if we can > > find the binary for all dependency it's probably best to link to them, > and > > let maven manage these dependencies. > > > > For 2, since the current CI is based on Jenkins, I imagine it would be > some > > sort of
Re: [DISCUSS] improve MXNet Scala release process
Hi all, about the credential management: We already have a solution based on restricted slaves [1] and AWS secrets manager [2] that is generally classified to generate binaries and handle credentials. It was designed with continuous deployment in mind, but we haven't tested it in that field yet. To properly assess the requirements, it would be great if we have this security critical part outlined for each release pipeline. We could then check and see if our existing solution matches all requirements or if further work is necessary. Best regards, Marco [1] https://cwiki.apache.org/confluence/display/MXNET/Restricted+jobs+and+nodes [2] https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html Naveen Swamy schrieb am Sa., 28. Juli 2018, 02:27: > I collaborate with Qing as a part of my day time job, to give you a little > more perspective on the proposed work > > For 1) > What we found is that users often run into conflicts when they use a > different version of the dependency(CUDA, CUDNN, OpenBLAS, OpenCV, etc,.) > and the one we build with MXNet backend and use in the MXNet Scala package. > Also it makes its not very straight-forward for users to install these > dependencies themselves in order to lower the entry barrier and to make > everything work out of the box we are thinking to build MXNet all these > dependencies with MXNet (as a static library) and embed them in the MXNet > Scala package. This is also inspired by the work you have done for Apache > MXNet pip packages, Ideally I would like to reuse some of that work. > > Maven does not manage the binaries, you still have to build the binary and > release them but what dependencies the binaries are built with is what > causes the confusion and problems. > In the past there were 20+ packages of MXNet and I reduced it to 3 (OSX, > Linux-CPU, Linux-GPU -- please see the discussion thread below), with the > latest version of the dependencies and we'll build/manage additional > packages based on the user demand. > > Please see the previous discussion about this topic. > > 1) Scala Maven Package discussion: > > https://lists.apache.org/thread.html/c3846515fc5560d826e7b6f47e9b8b6728a925e6f8decb9676456144@%3Cdev.mxnet.apache.org%3E > 2) Config for Scala Package: > > https://lists.apache.org/thread.html/0be6beb89cc2a792e7ba861a251f9a9a0b81fa36a5a0cd59d9f2cb6f@%3Cdev.mxnet.apache.org%3E > 3) Current Scala Package Release process: > > https://cwiki.apache.org/confluence/display/MXNET/MXNet-Scala+Release+Process > > > Hope that answers > > > On Fri, Jul 27, 2018 at 4:59 PM, Sheng Zha wrote: > > > Qing, > > > > For 1, why would it be a blocker, given that there were previous > releases? > > Has there been compatibility issues for scala packages? If so, why did we > > release? > > There are many maven packages that include binary already, so if we can > > find the binary for all dependency it's probably best to link to them, > and > > let maven manage these dependencies. > > > > For 2, since the current CI is based on Jenkins, I imagine it would be > some > > sort of Jenkins pipeline? Other people who're better versed in Jenkins > can > > chime in. > > > > Personally, I'm interested in 3 as well. I have a pipeline for building > pip > > packages that's currently not utilizing the CI, and the item 3 is the > > blocker too. Once you finish, it would be great to refer to the same > > solution, so that I can move it into the same CI. > > > > -sz > > > > On Fri, Jul 27, 2018 at 4:37 PM, Qing Lan wrote: > > > > > Hi all, > > > > > > Recently contributors on Scala Language development worked together and > > > finally able to publish Scala package on Maven. Now I would like to > > raise a > > > discussion to automate Scala release process and also discover a > standard > > > way to release different packages for MXNet so we won’t ask any > > individuals > > > to spend a long time to publish the package. > > > > > > There are three blocks that stop this automated process: > > > > > > 1. How to build general hardware-compatible backend dependencies for > > > MXNet (Linux CPU/GPU Mac OSX) > > > 2. How to automate the frontend release process and CI integration > > > 3. How to keep credentials for the release in the pipeline > > > > > > Scala Release process created by Naveen: https://cwiki.apache.org/ > > > confluence/display/MXNET/MXNet-Scala+Release+Process > > > > > > Thanks, > > > Qing > > > > > > > > >
Re: [DISCUSS] improve MXNet Scala release process
Thanks, Naveen. Once we have clarity on 3), it should be no problem for scala to reuse the same solution. For 1), if this is indeed an issue, it seems that we may have rushed a bit on the scala releases. Are there any user reports? -sz On Fri, Jul 27, 2018 at 5:26 PM, Naveen Swamy wrote: > I collaborate with Qing as a part of my day time job, to give you a little > more perspective on the proposed work > > For 1) > What we found is that users often run into conflicts when they use a > different version of the dependency(CUDA, CUDNN, OpenBLAS, OpenCV, etc,.) > and the one we build with MXNet backend and use in the MXNet Scala package. > Also it makes its not very straight-forward for users to install these > dependencies themselves in order to lower the entry barrier and to make > everything work out of the box we are thinking to build MXNet all these > dependencies with MXNet (as a static library) and embed them in the MXNet > Scala package. This is also inspired by the work you have done for Apache > MXNet pip packages, Ideally I would like to reuse some of that work. > > Maven does not manage the binaries, you still have to build the binary and > release them but what dependencies the binaries are built with is what > causes the confusion and problems. > In the past there were 20+ packages of MXNet and I reduced it to 3 (OSX, > Linux-CPU, Linux-GPU -- please see the discussion thread below), with the > latest version of the dependencies and we'll build/manage additional > packages based on the user demand. > > Please see the previous discussion about this topic. > > 1) Scala Maven Package discussion: > https://lists.apache.org/thread.html/c3846515fc5560d826e7b6f47e9b8b > 6728a925e6f8decb9676456144@%3Cdev.mxnet.apache.org%3E > 2) Config for Scala Package: > https://lists.apache.org/thread.html/0be6beb89cc2a792e7ba861a251f9a > 9a0b81fa36a5a0cd59d9f2cb6f@%3Cdev.mxnet.apache.org%3E > 3) Current Scala Package Release process: > https://cwiki.apache.org/confluence/display/MXNET/ > MXNet-Scala+Release+Process > > > Hope that answers > > > On Fri, Jul 27, 2018 at 4:59 PM, Sheng Zha wrote: > > > Qing, > > > > For 1, why would it be a blocker, given that there were previous > releases? > > Has there been compatibility issues for scala packages? If so, why did we > > release? > > There are many maven packages that include binary already, so if we can > > find the binary for all dependency it's probably best to link to them, > and > > let maven manage these dependencies. > > > > For 2, since the current CI is based on Jenkins, I imagine it would be > some > > sort of Jenkins pipeline? Other people who're better versed in Jenkins > can > > chime in. > > > > Personally, I'm interested in 3 as well. I have a pipeline for building > pip > > packages that's currently not utilizing the CI, and the item 3 is the > > blocker too. Once you finish, it would be great to refer to the same > > solution, so that I can move it into the same CI. > > > > -sz > > > > On Fri, Jul 27, 2018 at 4:37 PM, Qing Lan wrote: > > > > > Hi all, > > > > > > Recently contributors on Scala Language development worked together and > > > finally able to publish Scala package on Maven. Now I would like to > > raise a > > > discussion to automate Scala release process and also discover a > standard > > > way to release different packages for MXNet so we won’t ask any > > individuals > > > to spend a long time to publish the package. > > > > > > There are three blocks that stop this automated process: > > > > > > 1. How to build general hardware-compatible backend dependencies for > > > MXNet (Linux CPU/GPU Mac OSX) > > > 2. How to automate the frontend release process and CI integration > > > 3. How to keep credentials for the release in the pipeline > > > > > > Scala Release process created by Naveen: https://cwiki.apache.org/ > > > confluence/display/MXNET/MXNet-Scala+Release+Process > > > > > > Thanks, > > > Qing > > > > > > > > >
Re: [DISCUSS] improve MXNet Scala release process
I collaborate with Qing as a part of my day time job, to give you a little more perspective on the proposed work For 1) What we found is that users often run into conflicts when they use a different version of the dependency(CUDA, CUDNN, OpenBLAS, OpenCV, etc,.) and the one we build with MXNet backend and use in the MXNet Scala package. Also it makes its not very straight-forward for users to install these dependencies themselves in order to lower the entry barrier and to make everything work out of the box we are thinking to build MXNet all these dependencies with MXNet (as a static library) and embed them in the MXNet Scala package. This is also inspired by the work you have done for Apache MXNet pip packages, Ideally I would like to reuse some of that work. Maven does not manage the binaries, you still have to build the binary and release them but what dependencies the binaries are built with is what causes the confusion and problems. In the past there were 20+ packages of MXNet and I reduced it to 3 (OSX, Linux-CPU, Linux-GPU -- please see the discussion thread below), with the latest version of the dependencies and we'll build/manage additional packages based on the user demand. Please see the previous discussion about this topic. 1) Scala Maven Package discussion: https://lists.apache.org/thread.html/c3846515fc5560d826e7b6f47e9b8b6728a925e6f8decb9676456144@%3Cdev.mxnet.apache.org%3E 2) Config for Scala Package: https://lists.apache.org/thread.html/0be6beb89cc2a792e7ba861a251f9a9a0b81fa36a5a0cd59d9f2cb6f@%3Cdev.mxnet.apache.org%3E 3) Current Scala Package Release process: https://cwiki.apache.org/confluence/display/MXNET/MXNet-Scala+Release+Process Hope that answers On Fri, Jul 27, 2018 at 4:59 PM, Sheng Zha wrote: > Qing, > > For 1, why would it be a blocker, given that there were previous releases? > Has there been compatibility issues for scala packages? If so, why did we > release? > There are many maven packages that include binary already, so if we can > find the binary for all dependency it's probably best to link to them, and > let maven manage these dependencies. > > For 2, since the current CI is based on Jenkins, I imagine it would be some > sort of Jenkins pipeline? Other people who're better versed in Jenkins can > chime in. > > Personally, I'm interested in 3 as well. I have a pipeline for building pip > packages that's currently not utilizing the CI, and the item 3 is the > blocker too. Once you finish, it would be great to refer to the same > solution, so that I can move it into the same CI. > > -sz > > On Fri, Jul 27, 2018 at 4:37 PM, Qing Lan wrote: > > > Hi all, > > > > Recently contributors on Scala Language development worked together and > > finally able to publish Scala package on Maven. Now I would like to > raise a > > discussion to automate Scala release process and also discover a standard > > way to release different packages for MXNet so we won’t ask any > individuals > > to spend a long time to publish the package. > > > > There are three blocks that stop this automated process: > > > > 1. How to build general hardware-compatible backend dependencies for > > MXNet (Linux CPU/GPU Mac OSX) > > 2. How to automate the frontend release process and CI integration > > 3. How to keep credentials for the release in the pipeline > > > > Scala Release process created by Naveen: https://cwiki.apache.org/ > > confluence/display/MXNET/MXNet-Scala+Release+Process > > > > Thanks, > > Qing > > > > >
Re: [DISCUSS] improve MXNet Scala release process
Qing, For 1, why would it be a blocker, given that there were previous releases? Has there been compatibility issues for scala packages? If so, why did we release? There are many maven packages that include binary already, so if we can find the binary for all dependency it's probably best to link to them, and let maven manage these dependencies. For 2, since the current CI is based on Jenkins, I imagine it would be some sort of Jenkins pipeline? Other people who're better versed in Jenkins can chime in. Personally, I'm interested in 3 as well. I have a pipeline for building pip packages that's currently not utilizing the CI, and the item 3 is the blocker too. Once you finish, it would be great to refer to the same solution, so that I can move it into the same CI. -sz On Fri, Jul 27, 2018 at 4:37 PM, Qing Lan wrote: > Hi all, > > Recently contributors on Scala Language development worked together and > finally able to publish Scala package on Maven. Now I would like to raise a > discussion to automate Scala release process and also discover a standard > way to release different packages for MXNet so we won’t ask any individuals > to spend a long time to publish the package. > > There are three blocks that stop this automated process: > > 1. How to build general hardware-compatible backend dependencies for > MXNet (Linux CPU/GPU Mac OSX) > 2. How to automate the frontend release process and CI integration > 3. How to keep credentials for the release in the pipeline > > Scala Release process created by Naveen: https://cwiki.apache.org/ > confluence/display/MXNET/MXNet-Scala+Release+Process > > Thanks, > Qing > >
