perdasilva commented on issue #16202: [CD] Add COMMIT_ID param to release job URL: https://github.com/apache/incubator-mxnet/pull/16202#issuecomment-533264119 @aaronmarkham **Can you explain removing the git_sha param?** That was left-over from a previous CD PR. We originally had the functionality to clone a particular commit id, but then decided to remove it. It was too messy. I guess we forgot to remove everything... **Can you describe what steps to take when you do get stuck because of this change?** So, here I think it's important to first understand the source of the issue. The CD framework is made up of two Jenkins jobs: one that defines the overall CD pipeline (e.g. static lib release -> then, in parallel, pipelines for releasing the different artifacts (pypi, docker images, etc.) that depend on the static library). The other, is a parameterized job that executes the pipelines that build, validate, and publish a particular artifact (e.g. binary, pypi package, docker image, etc.). Essentially, the CD pipeline orchestrates calls to the parameterized job by calling it with the parameters that define the pipeline for each of the artifacts. Now, the CD pipeline is basically broken in two parts: the first builds and quality assures the libmxnet.so files and pushes them to the artifact repository **indexed by the commit id**. Then, the pipelines for each delivery channel take the binaries from the artifact repository, packages, tests and ships them. Our issue occurs if there are new commits to the branch from which that CD pipeline is being executed. That is, different (downstream) jobs could end up checking out different states of the repository, and there will be a failure because the binary for a commit id won't be found (since the libmxnet.so build pipeline worked off and got posted with a different commit id). This PR introduces the commit id parameter to the parameterized job to guarantee that the commit id used to checkout the CD pipeline will be used throughout the entire process. The only caveat here is that the interface, (i.e. the parameters) to the parameterized job is defined in the Jenkinsfile using pipeline syntax. This means, that every time that job runs, the current state (interface) could change depending on what is defined in the Jenkinsfile for the job at the supplied commit id. As an example, if we supply a commit id that is pre- adding the commit id parameter, if you try to re-execute the job, the parameter won't be there. This isn't a huge deal since this interface shouldn't change often at all. **Can you describe what steps to take when you do get stuck because of this change?** The job will need to be manually updated (by just running it and then stopping it). We could also possibly run a dummy parameterized job (that does nothing) as part of the CD pipeline to update the parameterized job's to that of the commit id being processed state. *Like if I were to try to emulate the process with a build.py run on a local instance, what would I need to do differently?* It doesn't quite translate to the process on the local instance because the binary is compiled there. Does any of this make sense? It's kinda convoluted to explain. I'll make a quick change and add this dummy run. Then we can avoid the problem altogether.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
