Having jenkins master migrated to new EC2 instance would need to have a backup of config.xml and jobs configuration. I can help to setup a new jenkins but a backup configuration is needed. Unfortunately, I don't have credential to login the current master and fetch a backup, do you have any suggestion, cos?
2015-05-25 18:14 GMT+08:00 Konstantin Boudnik <[email protected]>: > Hi Evans. > > This is great! Thank you very much for putting this together in time for > 1.0 > release! Ignite rc7 should get into the Incubator vote today, so in a > could of > days it will be ready. I will make a patch for the JIRA to fix it as well. > > I think the next step would be to move the master to an instance on new EC2 > account, followed by a request to INFRA to setup new ci.bigtop.apache.org > CNAME to point to that instance. > > I guess during that transition we'd need to clean up the old builds. For > Debian and Suse: Olaf and Peter would be the experts in these. Guys, could > you > please chime in? > > I will try to have RC1 this week (pending the Ignite vote) before > switching off > for a couple weeks of vacation. > > Thanks again! > Cos > On Thu, May 21, 2015 at 12:22AM, Evans Ye wrote: > > Folks, > > > > News on our Docker based CI. > > > > We now have nightly build for all the bigtop components available at > > http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/ > > And it's all green except the ignite-hadoop build. > > > > Those Docker prefixed jobs will run daily and fetch the newest code from > > master to build. > > This can give us a clear view to the status of Bigtop master code > > repository. > > For example, we can quickly know which commit breaks the build. > > > > What underlying supporting this feature are 4 EMR team donated EC2 > > instances and my docker images > > <https://github.com/evans-ye/bigtop-dockerfiles> for each Bigtop 1.0 > > targeted OS. > > > > To be notice that we only have centos 6, centos 7, ubuntu 14.04, and > fedora > > 20 builds available now, opensuse 13.2 and debian 8 are still not ready > > yet. As I mentioned in my previous mail, it would be great if someone > > familiar with the OS can help. :) > > > > Let's get back to the bigtop 1.0 release, with most of the CI > > infrastructure(slaves, images) are available and all the JIRAs that need > to > > be fixed in 1.0 are addressed, I believe we're close to the release. > > Now that the only blocker is ignite 1.1 (BIGTOP-1833 > > <https://issues.apache.org/jira/browse/BIGTOP-1833>), and the opensuse > 13.2 > > and debian 8 images. > > > > The next step of our CI improvement is to setup docker-provisioner and > runs > > smoke-tests to evaluate the system interoperability. I'll continue > working > > down this path. > > > > Anyone interesting in this or has any suggestion is welcomed as always. > > > > > > 2015-04-28 14:15 GMT+08:00 Konstantin Boudnik <[email protected]>: > > > > > ah, got it. Thanks for the explanations and all the work! > > > > > > On Tue, Apr 28, 2015 at 02:07PM, Evans Ye wrote: > > > > Hey Cos, > > > > > > > > I build new images only if needed. > > > > > > > > In 0.8, we have: > > > > > > > > centos-6.4 > > > > fedora-20 > > > > ubuntu-12.04 > > > > ubuntu-14.04 > > > > opensuse-12.3 > > > > > > > > Compare with the 1.0 BOM: > > > > > > > > OS: > > > > CentOS6, CentOS7, Fedora 20 > > > > SLES 11SP3, SLES12, OpenSUSE 13.2 > > > > Ubuntu 14.04 LTS, Debian 8 > > > > > > > > > > > > fedora-20 and ubuntu-14.04 are still roman's images in 1.0. > > > > > > > > But centos-6.4 is quite old, hence I build a new one based on > centos-6.6. > > > > > > > > centos-7 and debian are new comers in 1.0, so we have to create them > from > > > > scratch. > > > > > > > > opensuse has been upgraded from 12.3 to 13.2. We need a new image to > be > > > > built as well. > > > > > > > > Evans > > > > > > > > 2015-04-28 3:04 GMT+08:00 Konstantin Boudnik <[email protected]>: > > > > > > > > > That's great! Thank you Evans! > > > > > > > > > > I am sure Peter Linnell should be able to give you a hand with > *Suse > > > > > platform. > > > > > Olaf, could you please help Evans with Debian? > > > > > > > > > > One question about dockers: any reason you've chosen not to use the > > > images > > > > > that Roman has put together initially? Not like I care one way or > > > another, > > > > > but > > > > > I am using those at the moment and want to make sure that we are > > > building > > > > > in > > > > > the same environment across the board. > > > > > > > > > > Also, per Nate's request I have added S3 policy to the bigtop-ci > group > > > on > > > > > AWS, > > > > > so you guys should be able to use S3 if needed. > > > > > > > > > > Thanks, > > > > > Cos > > > > > > > > > > On Tue, Apr 28, 2015 at 02:25AM, Evans Ye wrote: > > > > > > Hi folks, > > > > > > > > > > > > Allow me to report some progress on our CI settings for 1.0 > release. > > > > > > ([email protected] asked me about the status and would like to > help > > > > > > but sorry I didn't respond you well back then) > > > > > > > > > > > > > > > > > > For 1.0 to release, we need to build bigtop by those supported OS > > > listed > > > > > in > > > > > > BIGTOP-1480. > > > > > > > > > > > > And I plan to use use docker images to build 1.0 release > directly so > > > that > > > > > > we don't need to setup jenkins slaves one-by-one. (the fastest > way I > > > can > > > > > > think of for speeding up 1.0 release) > > > > > > > > > > > > My plan is to prepare bigtop/puppet images by ourselves > > > > > > <https://github.com/evans-ye/bigtop-dockerfiles>and apply > > > > > bigtop_toolchain > > > > > > by jenkins > > > > > > < > http://bigtop01.cloudera.org:8080/view/Docker/job/Docker-Toolchain/ > > > >to > > > > > get > > > > > > bigtop/slaves images. > > > > > > > > > > > > I've built and tested some of the OS in bigtop/puppet family > (stored > > > > > > as evansye/puppet > > > > > > <https://registry.hub.docker.com/u/evansye/puppet/tags/manage/ > > > > > >temporary), > > > > > > but some of the OS do not work properly since I'm lacking of > > > knowledge on > > > > > > them. > > > > > > > > > > > > ***worked*** > > > > > > > > > > > > centos-6 > > > > > > centos-7 > > > > > > ubuntu-14.04 (unchanged) > > > > > > fedora-20 (unchanged) > > > > > > > > > > > > ***not working*** > > > > > > opensuse-13.2 > > > > > > ---> I can't have "hostname" package installed, it seems that is > > > needed > > > > > by > > > > > > puppet to determine what node to apply > > > > > > debian 8 > > > > > > ---> no wget and curl available > > > > > > [root@bigtop ~]# $ docker run -ti --rm debian:8 bash -c > 'apt-get -y > > > > > install > > > > > > curl wget' > > > > > > Reading package lists... Done > > > > > > Building dependency tree... Done > > > > > > E: Unable to locate package curl > > > > > > E: Unable to locate package wget > > > > > > > > > > > > > > > > > > Todo list: > > > > > > 1. It would be a great help if opensuse and debian expert can > come to > > > > > help > > > > > > on those issues. > > > > > > 2. I'd like to have the credential for bigtop dockerhub so that I > > > can put > > > > > > new OS images on it. > > > > > > 3. Have all package in trunk built by new Docker slaves > > > > > > > > > > > > Maybe Roman can help me out on *#2. :)* > > > > > > You can enclose me the credential by my pgp: > > > > > > curl http://people.apache.org/~evansye/evansye.pgp > > > > > > > > > > > > That it. Suggestions are welcome! > > > > > > > > > > > > Thanks, > > > > > > Evans > > > > > > > > > > > > 2015-03-31 7:09 GMT+08:00 Konstantin Boudnik <[email protected]>: > > > > > > > > > > > > > Currently all the slaves shut-down: looks like the instances > have > > > been > > > > > > > turned > > > > > > > off as I can not relaunch the slaves. Will try to restart them > > > tonight > > > > > > > once I > > > > > > > am on machine with AWS keys. > > > > > > > > > > > > > > Cos > > > > > > > > > > > > > > On Tue, Mar 24, 2015 at 06:06AM, Konstantin Boudnik wrote: > > > > > > > > Guys, > > > > > > > > > > > > > > > > I want to start a separate thread to track the CI > preparations > > > for > > > > > the > > > > > > > release > > > > > > > > next month (fingers crossed). Clearly, we can make a release > > > without > > > > > CI, > > > > > > > but > > > > > > > > it'd way easier to test and create binary artifacts if we > have a > > > > > working > > > > > > > > environment for official validation. Roman has done a lot in > this > > > > > > > direction > > > > > > > > (many thanks!), but there are still a few rough edges, which > > > might be > > > > > > > easy to > > > > > > > > finish of. > > > > > > > > > > > > > > > > I want to figure out a couple of things: > > > > > > > > - what's the state of CI and how much still needs to be done > > > (Rvs? > > > > > > > Could you > > > > > > > > share any first hand feedback?) > > > > > > > > - who would be able to help with the CI completion? I can > commit > > > > > some > > > > > > > of my > > > > > > > > cycles, but it'd be great to have few more hands on that. > > > Clearly, > > > > > > > some > > > > > > > > Jenkins-foo and prior CI skills won't hurt ;) > > > > > > > > > > > > > > > > Please chime in if you can help. Thanks a lot! > > > > > > > > Cos > > > > > > > > > > > > > > > > > > > > > > > >
