Hi Adam It completely makes sense. Would you like to take a look at what we can do with github actions? We already have set up actions in yunikorn-core and yunikorn-k8shim to execute UTs, we need that to add web and interface repos too; also it will be good to know what it can help with the release, e.g automatically build source code tarball for us.
Weiwei On Thu, Apr 16, 2020 at 5:39 AM Adam Antal <[email protected]> wrote: > Hi everyone, > > Just my two cents here... > > I think the tendency nowadays is to add these tools to build the code to > the repository itself. See Gitlab yamls or Github action file that can > declare the compile+test phase for the CI. > > Gitlab docs perfectly sums up why I think it can be a better choice to > include it next to the code: > > > Because .gitlab-ci.yml is in the repository and is version controlled, > old > > versions still build successfully, forks can easily make use of CI, > > branches can have different pipelines and jobs, and you have a single > > source of truth for CI. You can read more about the reasons why we are > > using .gitlab-ci.yml in our blog about it > > < > https://about.gitlab.com/blog/2015/05/06/why-were-replacing-gitlab-ci-jobs-with-gitlab-ci-dot-yml/ > > > > . > > > Also I recommend reading the article. > > Regards, > Adam > > On Wed, Apr 15, 2020 at 11:59 PM Weiwei Yang <[email protected]> wrote: > > > Hi Wilfred/Sunil > > > > Thanks for the feedback. > > For a multi-repo project like YuniKorn, I think it makes sense to have a > > separate one to manage them. > > Put release things (script, doc, files) in any of the source code repo > is a > > bit weird to me. I have already created > > https://github.com/apache/incubator-yunikorn-release/. *which is for the > > release managers only.* This can avoid further confusion and also help us > > to do future releases in minutes. > > The docker file approach, which provides a build env without pollute > > people's local env, this might be helpful in certain cases. Such as when > > you need to install a lot of binaries to make a build. For YuniKorn, > since > > we just rely on Go for the build env, I don't see how useful it is at > > present. I am open to adopting that approach as long as it helps. > > > > Weiwei > > > > > > On Wed, Apr 15, 2020 at 8:06 AM Sunil Govindan <[email protected]> > wrote: > > > > > Thanks Weiwei and Wilfred. > > > > > > +0 on this approach. I am overall feeling fine with the approach > > suggested. > > > However I may have personally prefer to do this in a create-release.sh > > and > > > Dockerfile like hadoop model. > > > > > > As Weiwei pointed out, its used in Opwnwhisk pjct as well, may be we > can > > > try this out and find the positiveness. > > > > > > I think, we should make the scripts or configs in this repo more > dynamic > > in > > > nature agnostic to any release version. > > > RM can supply this version etc as a run time param to these scripts. > > > > > > Thanks > > > Sunil > > > > > > On Tue, Apr 14, 2020 at 12:26 PM Weiwei Yang <[email protected]> > > wrote: > > > > > > > Hi Wilfred > > > > > > > > Thanks for the feedback. > > > > > > > > The reason to introduce a new repo is to have some tools/docs easier > > for > > > > release managers. I found this is a good practice when you have > > multiple > > > > repos. Please see Apache OpenWhisk as an example. > > > > > > > > If you look at the config file I shared, that helps to track source > > code > > > > revision of different repos, which is easy to assemble a release. > > > > > > > > I propose to do unified release, which means we alway release all > repos > > > > together. This is simpler to manage. > > > > > > > > If you look at the archive file, I should have put a script and a > > README > > > > about how to build docker images. Can you please take a look at that? > > If > > > > that is not there, pls see the script on github. > > > > > > > > Thanks for verifying the content. Could you please let me know what > > files > > > > are wrong? > > > > > > > > > > > > > > > > -- > > > > Weiwei > > > > On Apr 13, 2020, 11:30 PM -0700, Wilfred Spiegelenburg < > > > > [email protected]>, wrote: > > > > > Hi Weiwei, > > > > > > > > > > Do we really need another repo for three files? > > > > > We should sleep track of this in the core repo not in another repo > > > which > > > > we > > > > > need to release manage again. I think managing the release from the > > > core > > > > > repo will make it easier later on if we need to or want to change > the > > > > build > > > > > process further. Now we need need to manage and track soo many > repos > > > that > > > > > it becomes more and more difficult. > > > > > We also need to keep in mind that version information is in the > > module > > > > > files. There might thus be more that needs to change for a release. > > The > > > > > other thing is that we might not want to release a new version of > one > > > of > > > > > the components while updating another component. That would means > > that > > > we > > > > > need to release manage 5 repositories for one release, including > all > > > the > > > > > overhead. > > > > > > > > > > Apache releases are source releases. We still need to provide some > > kind > > > > of > > > > > make etc over the source code also. I agree with the fact that we > > need > > > to > > > > > provide one source archive that is signed. However with the current > > > build > > > > > process just the k8shim code is enough to build the docker image. > The > > > > other > > > > > code repos will be pulled in from github. The mod file point there > > for > > > > all > > > > > go dependencies including the core and SI. It does not provide any > > > detail > > > > > on the how and what for any of the repos. We need to provide some > > build > > > > > instructions in the root of the source archive. I would not know > how > > to > > > > > build from the source package if we just add the checked out code > > into > > > > it. > > > > > We need to provide some steps even if they are just pointers to > > > existing > > > > > docs. > > > > > I also don't think we have the correct files in the archive with > the > > > > > current generated archive. > > > > > > > > > > Wilfred > > > > > > > > > > > > > > > > On Mon, 13 Apr 2020 at 14:01, Weiwei Yang <[email protected]> > wrote: > > > > > > > > > > > For 0.8 release, I did some work and I wanted to share the latest > > > > status. I > > > > > > think we should target for *docker-image-based* release mode. I > > > > propose to > > > > > > release a unified open-source tarball, we don't release a binary > > > > tarball > > > > > > (not a must [1]). Things have been *DONE*: > > > > > > > > > > > > 1. I have created a repo for release mgmt: > > > > > > https://github.com/yangwwei/yunikorn-release, I think we need to > > > move > > > > > > this to apache repo too. > > > > > > > > > > > 2. This repo has the instructions and tools for a release. The tool > > > > > > loads configs from > > > > > > > > > > > > > > > > > > > > > > https://github.com/yangwwei/yunikorn-release/blob/master/tools/release-configs.json > > > > > > and > > > > > > downloads source code from certain repo/branch/hash to assemble > the > > > > > > release > > > > > > artifacts > > > > > > > > > > 3. This repo contains a *build-docker-image.sh* to build yunikorn > > > docker > > > > > > images (scheduler, admission-controller, and web) > > > > > > 4. I have created *branch-0.8* for all 4 repos > > > > > > 5. The generated tarball will also have the helm chart for user > to > > > > > > install and run yunikorn on an existing K8s cluster > > > > > > 6. I tried to generate PGP key and sign the tarball > > > > > > > > > > > > Things *TODO* > > > > > > > > > > > > 1. Create a repo for yunikorn-release under ASF > > > > > > 2. IIUC, https://issues.apache.org/jira/browse/YUNIKORN-79 is a > > > > blocker > > > > > > for 0.8. Can we get this fixed ASAP? > > > > > > 3. Once #2 is done, create a tag for 0.8-rc1 and start the voting > > > > thread > > > > > > > > > > > > Thanks! > > > > > > > > > > > > [1] https://infra.apache.org/release-publishing.html *The Apache > > > > Software > > > > > > Foundation exists to create open source software, so the > > fundamental > > > > > > requirement for a release is that it has the necessary source > code > > to > > > > build > > > > > > the project. A project may provide compiled binaries of each > > release > > > > for > > > > > > the convenience of users.* > > > > > > > > > > > > > > > >
