Hi Evans. This is great! Thank you very much for putting this together in time for 1.0 release! Ignite rc7 should get into the Incubator vote today, so in a could of days it will be ready. I will make a patch for the JIRA to fix it as well.
I think the next step would be to move the master to an instance on new EC2 account, followed by a request to INFRA to setup new ci.bigtop.apache.org CNAME to point to that instance. I guess during that transition we'd need to clean up the old builds. For Debian and Suse: Olaf and Peter would be the experts in these. Guys, could you please chime in? I will try to have RC1 this week (pending the Ignite vote) before switching off for a couple weeks of vacation. Thanks again! Cos On Thu, May 21, 2015 at 12:22AM, Evans Ye wrote: > Folks, > > News on our Docker based CI. > > We now have nightly build for all the bigtop components available at > http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/ > And it's all green except the ignite-hadoop build. > > Those Docker prefixed jobs will run daily and fetch the newest code from > master to build. > This can give us a clear view to the status of Bigtop master code > repository. > For example, we can quickly know which commit breaks the build. > > What underlying supporting this feature are 4 EMR team donated EC2 > instances and my docker images > <https://github.com/evans-ye/bigtop-dockerfiles> for each Bigtop 1.0 > targeted OS. > > To be notice that we only have centos 6, centos 7, ubuntu 14.04, and fedora > 20 builds available now, opensuse 13.2 and debian 8 are still not ready > yet. As I mentioned in my previous mail, it would be great if someone > familiar with the OS can help. :) > > Let's get back to the bigtop 1.0 release, with most of the CI > infrastructure(slaves, images) are available and all the JIRAs that need to > be fixed in 1.0 are addressed, I believe we're close to the release. > Now that the only blocker is ignite 1.1 (BIGTOP-1833 > <https://issues.apache.org/jira/browse/BIGTOP-1833>), and the opensuse 13.2 > and debian 8 images. > > The next step of our CI improvement is to setup docker-provisioner and runs > smoke-tests to evaluate the system interoperability. I'll continue working > down this path. > > Anyone interesting in this or has any suggestion is welcomed as always. > > > 2015-04-28 14:15 GMT+08:00 Konstantin Boudnik <[email protected]>: > > > ah, got it. Thanks for the explanations and all the work! > > > > On Tue, Apr 28, 2015 at 02:07PM, Evans Ye wrote: > > > Hey Cos, > > > > > > I build new images only if needed. > > > > > > In 0.8, we have: > > > > > > centos-6.4 > > > fedora-20 > > > ubuntu-12.04 > > > ubuntu-14.04 > > > opensuse-12.3 > > > > > > Compare with the 1.0 BOM: > > > > > > OS: > > > CentOS6, CentOS7, Fedora 20 > > > SLES 11SP3, SLES12, OpenSUSE 13.2 > > > Ubuntu 14.04 LTS, Debian 8 > > > > > > > > > fedora-20 and ubuntu-14.04 are still roman's images in 1.0. > > > > > > But centos-6.4 is quite old, hence I build a new one based on centos-6.6. > > > > > > centos-7 and debian are new comers in 1.0, so we have to create them from > > > scratch. > > > > > > opensuse has been upgraded from 12.3 to 13.2. We need a new image to be > > > built as well. > > > > > > Evans > > > > > > 2015-04-28 3:04 GMT+08:00 Konstantin Boudnik <[email protected]>: > > > > > > > That's great! Thank you Evans! > > > > > > > > I am sure Peter Linnell should be able to give you a hand with *Suse > > > > platform. > > > > Olaf, could you please help Evans with Debian? > > > > > > > > One question about dockers: any reason you've chosen not to use the > > images > > > > that Roman has put together initially? Not like I care one way or > > another, > > > > but > > > > I am using those at the moment and want to make sure that we are > > building > > > > in > > > > the same environment across the board. > > > > > > > > Also, per Nate's request I have added S3 policy to the bigtop-ci group > > on > > > > AWS, > > > > so you guys should be able to use S3 if needed. > > > > > > > > Thanks, > > > > Cos > > > > > > > > On Tue, Apr 28, 2015 at 02:25AM, Evans Ye wrote: > > > > > Hi folks, > > > > > > > > > > Allow me to report some progress on our CI settings for 1.0 release. > > > > > ([email protected] asked me about the status and would like to help > > > > > but sorry I didn't respond you well back then) > > > > > > > > > > > > > > > For 1.0 to release, we need to build bigtop by those supported OS > > listed > > > > in > > > > > BIGTOP-1480. > > > > > > > > > > And I plan to use use docker images to build 1.0 release directly so > > that > > > > > we don't need to setup jenkins slaves one-by-one. (the fastest way I > > can > > > > > think of for speeding up 1.0 release) > > > > > > > > > > My plan is to prepare bigtop/puppet images by ourselves > > > > > <https://github.com/evans-ye/bigtop-dockerfiles>and apply > > > > bigtop_toolchain > > > > > by jenkins > > > > > <http://bigtop01.cloudera.org:8080/view/Docker/job/Docker-Toolchain/ > > >to > > > > get > > > > > bigtop/slaves images. > > > > > > > > > > I've built and tested some of the OS in bigtop/puppet family (stored > > > > > as evansye/puppet > > > > > <https://registry.hub.docker.com/u/evansye/puppet/tags/manage/ > > > > >temporary), > > > > > but some of the OS do not work properly since I'm lacking of > > knowledge on > > > > > them. > > > > > > > > > > ***worked*** > > > > > > > > > > centos-6 > > > > > centos-7 > > > > > ubuntu-14.04 (unchanged) > > > > > fedora-20 (unchanged) > > > > > > > > > > ***not working*** > > > > > opensuse-13.2 > > > > > ---> I can't have "hostname" package installed, it seems that is > > needed > > > > by > > > > > puppet to determine what node to apply > > > > > debian 8 > > > > > ---> no wget and curl available > > > > > [root@bigtop ~]# $ docker run -ti --rm debian:8 bash -c 'apt-get -y > > > > install > > > > > curl wget' > > > > > Reading package lists... Done > > > > > Building dependency tree... Done > > > > > E: Unable to locate package curl > > > > > E: Unable to locate package wget > > > > > > > > > > > > > > > Todo list: > > > > > 1. It would be a great help if opensuse and debian expert can come to > > > > help > > > > > on those issues. > > > > > 2. I'd like to have the credential for bigtop dockerhub so that I > > can put > > > > > new OS images on it. > > > > > 3. Have all package in trunk built by new Docker slaves > > > > > > > > > > Maybe Roman can help me out on *#2. :)* > > > > > You can enclose me the credential by my pgp: > > > > > curl http://people.apache.org/~evansye/evansye.pgp > > > > > > > > > > That it. Suggestions are welcome! > > > > > > > > > > Thanks, > > > > > Evans > > > > > > > > > > 2015-03-31 7:09 GMT+08:00 Konstantin Boudnik <[email protected]>: > > > > > > > > > > > Currently all the slaves shut-down: looks like the instances have > > been > > > > > > turned > > > > > > off as I can not relaunch the slaves. Will try to restart them > > tonight > > > > > > once I > > > > > > am on machine with AWS keys. > > > > > > > > > > > > Cos > > > > > > > > > > > > On Tue, Mar 24, 2015 at 06:06AM, Konstantin Boudnik wrote: > > > > > > > Guys, > > > > > > > > > > > > > > I want to start a separate thread to track the CI preparations > > for > > > > the > > > > > > release > > > > > > > next month (fingers crossed). Clearly, we can make a release > > without > > > > CI, > > > > > > but > > > > > > > it'd way easier to test and create binary artifacts if we have a > > > > working > > > > > > > environment for official validation. Roman has done a lot in this > > > > > > direction > > > > > > > (many thanks!), but there are still a few rough edges, which > > might be > > > > > > easy to > > > > > > > finish of. > > > > > > > > > > > > > > I want to figure out a couple of things: > > > > > > > - what's the state of CI and how much still needs to be done > > (Rvs? > > > > > > Could you > > > > > > > share any first hand feedback?) > > > > > > > - who would be able to help with the CI completion? I can commit > > > > some > > > > > > of my > > > > > > > cycles, but it'd be great to have few more hands on that. > > Clearly, > > > > > > some > > > > > > > Jenkins-foo and prior CI skills won't hurt ;) > > > > > > > > > > > > > > Please chime in if you can help. Thanks a lot! > > > > > > > Cos > > > > > > > > > > > > > > > > > > >
signature.asc
Description: Digital signature
