Let me get back to you on this tmr. Ok?
On Mon, May 25, 2015 at 1:36 PM, Konstantin Boudnik <[email protected]> wrote: > On Tue, May 26, 2015 at 01:37AM, Evans Ye wrote: >> Having jenkins master migrated to new EC2 instance would need to have a >> backup of config.xml and jobs configuration. I can help to setup a new >> jenkins but a backup configuration is needed. Unfortunately, I don't have >> credential to login the current master and fetch a backup, do you have any >> suggestion, cos? > > I know that Roman has all backups as he has dealt with the original setup and > restored it a couple of times after recent crashes. I will ping him off-line > to make sure he's getting the message ;) > > Thanks! > Cos > >> >> 2015-05-25 18:14 GMT+08:00 Konstantin Boudnik <[email protected]>: >> >> > Hi Evans. >> > >> > This is great! Thank you very much for putting this together in time for >> > 1.0 >> > release! Ignite rc7 should get into the Incubator vote today, so in a >> > could of >> > days it will be ready. I will make a patch for the JIRA to fix it as well. >> > >> > I think the next step would be to move the master to an instance on new EC2 >> > account, followed by a request to INFRA to setup new ci.bigtop.apache.org >> > CNAME to point to that instance. >> > >> > I guess during that transition we'd need to clean up the old builds. For >> > Debian and Suse: Olaf and Peter would be the experts in these. Guys, could >> > you >> > please chime in? >> > >> > I will try to have RC1 this week (pending the Ignite vote) before >> > switching off >> > for a couple weeks of vacation. >> > >> > Thanks again! >> > Cos >> > On Thu, May 21, 2015 at 12:22AM, Evans Ye wrote: >> > > Folks, >> > > >> > > News on our Docker based CI. >> > > >> > > We now have nightly build for all the bigtop components available at >> > > http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/ >> > > And it's all green except the ignite-hadoop build. >> > > >> > > Those Docker prefixed jobs will run daily and fetch the newest code from >> > > master to build. >> > > This can give us a clear view to the status of Bigtop master code >> > > repository. >> > > For example, we can quickly know which commit breaks the build. >> > > >> > > What underlying supporting this feature are 4 EMR team donated EC2 >> > > instances and my docker images >> > > <https://github.com/evans-ye/bigtop-dockerfiles> for each Bigtop 1.0 >> > > targeted OS. >> > > >> > > To be notice that we only have centos 6, centos 7, ubuntu 14.04, and >> > fedora >> > > 20 builds available now, opensuse 13.2 and debian 8 are still not ready >> > > yet. As I mentioned in my previous mail, it would be great if someone >> > > familiar with the OS can help. :) >> > > >> > > Let's get back to the bigtop 1.0 release, with most of the CI >> > > infrastructure(slaves, images) are available and all the JIRAs that need >> > to >> > > be fixed in 1.0 are addressed, I believe we're close to the release. >> > > Now that the only blocker is ignite 1.1 (BIGTOP-1833 >> > > <https://issues.apache.org/jira/browse/BIGTOP-1833>), and the opensuse >> > 13.2 >> > > and debian 8 images. >> > > >> > > The next step of our CI improvement is to setup docker-provisioner and >> > runs >> > > smoke-tests to evaluate the system interoperability. I'll continue >> > working >> > > down this path. >> > > >> > > Anyone interesting in this or has any suggestion is welcomed as always. >> > > >> > > >> > > 2015-04-28 14:15 GMT+08:00 Konstantin Boudnik <[email protected]>: >> > > >> > > > ah, got it. Thanks for the explanations and all the work! >> > > > >> > > > On Tue, Apr 28, 2015 at 02:07PM, Evans Ye wrote: >> > > > > Hey Cos, >> > > > > >> > > > > I build new images only if needed. >> > > > > >> > > > > In 0.8, we have: >> > > > > >> > > > > centos-6.4 >> > > > > fedora-20 >> > > > > ubuntu-12.04 >> > > > > ubuntu-14.04 >> > > > > opensuse-12.3 >> > > > > >> > > > > Compare with the 1.0 BOM: >> > > > > >> > > > > OS: >> > > > > CentOS6, CentOS7, Fedora 20 >> > > > > SLES 11SP3, SLES12, OpenSUSE 13.2 >> > > > > Ubuntu 14.04 LTS, Debian 8 >> > > > > >> > > > > >> > > > > fedora-20 and ubuntu-14.04 are still roman's images in 1.0. >> > > > > >> > > > > But centos-6.4 is quite old, hence I build a new one based on >> > centos-6.6. >> > > > > >> > > > > centos-7 and debian are new comers in 1.0, so we have to create them >> > from >> > > > > scratch. >> > > > > >> > > > > opensuse has been upgraded from 12.3 to 13.2. We need a new image to >> > be >> > > > > built as well. >> > > > > >> > > > > Evans >> > > > > >> > > > > 2015-04-28 3:04 GMT+08:00 Konstantin Boudnik <[email protected]>: >> > > > > >> > > > > > That's great! Thank you Evans! >> > > > > > >> > > > > > I am sure Peter Linnell should be able to give you a hand with >> > *Suse >> > > > > > platform. >> > > > > > Olaf, could you please help Evans with Debian? >> > > > > > >> > > > > > One question about dockers: any reason you've chosen not to use the >> > > > images >> > > > > > that Roman has put together initially? Not like I care one way or >> > > > another, >> > > > > > but >> > > > > > I am using those at the moment and want to make sure that we are >> > > > building >> > > > > > in >> > > > > > the same environment across the board. >> > > > > > >> > > > > > Also, per Nate's request I have added S3 policy to the bigtop-ci >> > group >> > > > on >> > > > > > AWS, >> > > > > > so you guys should be able to use S3 if needed. >> > > > > > >> > > > > > Thanks, >> > > > > > Cos >> > > > > > >> > > > > > On Tue, Apr 28, 2015 at 02:25AM, Evans Ye wrote: >> > > > > > > Hi folks, >> > > > > > > >> > > > > > > Allow me to report some progress on our CI settings for 1.0 >> > release. >> > > > > > > ([email protected] asked me about the status and would like to >> > help >> > > > > > > but sorry I didn't respond you well back then) >> > > > > > > >> > > > > > > >> > > > > > > For 1.0 to release, we need to build bigtop by those supported OS >> > > > listed >> > > > > > in >> > > > > > > BIGTOP-1480. >> > > > > > > >> > > > > > > And I plan to use use docker images to build 1.0 release >> > directly so >> > > > that >> > > > > > > we don't need to setup jenkins slaves one-by-one. (the fastest >> > way I >> > > > can >> > > > > > > think of for speeding up 1.0 release) >> > > > > > > >> > > > > > > My plan is to prepare bigtop/puppet images by ourselves >> > > > > > > <https://github.com/evans-ye/bigtop-dockerfiles>and apply >> > > > > > bigtop_toolchain >> > > > > > > by jenkins >> > > > > > > < >> > http://bigtop01.cloudera.org:8080/view/Docker/job/Docker-Toolchain/ >> > > > >to >> > > > > > get >> > > > > > > bigtop/slaves images. >> > > > > > > >> > > > > > > I've built and tested some of the OS in bigtop/puppet family >> > (stored >> > > > > > > as evansye/puppet >> > > > > > > <https://registry.hub.docker.com/u/evansye/puppet/tags/manage/ >> > > > > > >temporary), >> > > > > > > but some of the OS do not work properly since I'm lacking of >> > > > knowledge on >> > > > > > > them. >> > > > > > > >> > > > > > > ***worked*** >> > > > > > > >> > > > > > > centos-6 >> > > > > > > centos-7 >> > > > > > > ubuntu-14.04 (unchanged) >> > > > > > > fedora-20 (unchanged) >> > > > > > > >> > > > > > > ***not working*** >> > > > > > > opensuse-13.2 >> > > > > > > ---> I can't have "hostname" package installed, it seems that is >> > > > needed >> > > > > > by >> > > > > > > puppet to determine what node to apply >> > > > > > > debian 8 >> > > > > > > ---> no wget and curl available >> > > > > > > [root@bigtop ~]# $ docker run -ti --rm debian:8 bash -c >> > 'apt-get -y >> > > > > > install >> > > > > > > curl wget' >> > > > > > > Reading package lists... Done >> > > > > > > Building dependency tree... Done >> > > > > > > E: Unable to locate package curl >> > > > > > > E: Unable to locate package wget >> > > > > > > >> > > > > > > >> > > > > > > Todo list: >> > > > > > > 1. It would be a great help if opensuse and debian expert can >> > come to >> > > > > > help >> > > > > > > on those issues. >> > > > > > > 2. I'd like to have the credential for bigtop dockerhub so that I >> > > > can put >> > > > > > > new OS images on it. >> > > > > > > 3. Have all package in trunk built by new Docker slaves >> > > > > > > >> > > > > > > Maybe Roman can help me out on *#2. :)* >> > > > > > > You can enclose me the credential by my pgp: >> > > > > > > curl http://people.apache.org/~evansye/evansye.pgp >> > > > > > > >> > > > > > > That it. Suggestions are welcome! >> > > > > > > >> > > > > > > Thanks, >> > > > > > > Evans >> > > > > > > >> > > > > > > 2015-03-31 7:09 GMT+08:00 Konstantin Boudnik <[email protected]>: >> > > > > > > >> > > > > > > > Currently all the slaves shut-down: looks like the instances >> > have >> > > > been >> > > > > > > > turned >> > > > > > > > off as I can not relaunch the slaves. Will try to restart them >> > > > tonight >> > > > > > > > once I >> > > > > > > > am on machine with AWS keys. >> > > > > > > > >> > > > > > > > Cos >> > > > > > > > >> > > > > > > > On Tue, Mar 24, 2015 at 06:06AM, Konstantin Boudnik wrote: >> > > > > > > > > Guys, >> > > > > > > > > >> > > > > > > > > I want to start a separate thread to track the CI >> > preparations >> > > > for >> > > > > > the >> > > > > > > > release >> > > > > > > > > next month (fingers crossed). Clearly, we can make a release >> > > > without >> > > > > > CI, >> > > > > > > > but >> > > > > > > > > it'd way easier to test and create binary artifacts if we >> > have a >> > > > > > working >> > > > > > > > > environment for official validation. Roman has done a lot in >> > this >> > > > > > > > direction >> > > > > > > > > (many thanks!), but there are still a few rough edges, which >> > > > might be >> > > > > > > > easy to >> > > > > > > > > finish of. >> > > > > > > > > >> > > > > > > > > I want to figure out a couple of things: >> > > > > > > > > - what's the state of CI and how much still needs to be done >> > > > (Rvs? >> > > > > > > > Could you >> > > > > > > > > share any first hand feedback?) >> > > > > > > > > - who would be able to help with the CI completion? I can >> > commit >> > > > > > some >> > > > > > > > of my >> > > > > > > > > cycles, but it'd be great to have few more hands on that. >> > > > Clearly, >> > > > > > > > some >> > > > > > > > > Jenkins-foo and prior CI skills won't hurt ;) >> > > > > > > > > >> > > > > > > > > Please chime in if you can help. Thanks a lot! >> > > > > > > > > Cos >> > > > > > > > > >> > > > > > > > >> > > > > > >> > > > >> >
