Idea would like to work through with the group is post 1.0, evolving the CI and project process to be able to do real-time, or just-in-time builds.
We could have a "bigtop core", which would be components like Hadoop/hdfs/zookeeper/etc. Thinks like hbase, hive, etc would be next level components, then things like pig, cascading, etc would be the real fast movers. Think another area we can investigate is components that want to integrate with bigtop but are not dependent on jvm stack and just the interfaces/api's of the core components. Seems those as well could be fast movers Don’t know if that answers your question or not Andre, hopefully with your guys help we can get some Cascading action going and get the build/release process evolving so no matter how fast you guys are releasing users can deploy the bits integrated fashion to other Bigtop components -----Original Message----- From: Evans Ye [mailto:[email protected]] Sent: Tuesday, May 12, 2015 9:16 AM To: [email protected] Subject: Re: ci infrastructure and release process for Cascading integration Hi Andre I take you're not asking technical questions but rather than wondering something related to the process or policy. AFAIK, at this moment we don't have a process to have the up-to-date packages available. (The Bigtop CI used to be working, but collapsed when most of the build slaves went out of available) So in your example, when upgrading upstream component version from 1.1 to 2.0, they would need to submit a patch and build the package by themselves. But, thanks to the EMR team sponsoring us EC2 resources, we're now rebuilding our CI infra, which surely would have nightly build(RPM/DEB) available. OTOH, I can recall that Nate is also working on bigtop repo nightly build, when that comes online we can always have SNAPSHOT repos available as well. Evans 2015-05-12 0:27 GMT+08:00 Andre Kelpe <[email protected]>: > If that is true, what would happen in the case where we want to make a > new major release available? Suppose we have project in 1.x and now as > a upstream we release 2.0. Would such a change still be covered by > this? How would somebody trying to rebuild all packages handle these > changes? If we submit a package update, the bigtop source release will > only have the spec files/debian directory from the time that a bigtop > release was made. Does that still work with the apache rules? > > - André > > On Mon, May 11, 2015 at 3:52 PM, jay vyas > <[email protected]> > wrote: > > > i think -- that -- since bigtop RPM/DEB packages ARE NOT bound by > > the > ASF > > guidelines, they are just binary convenience artifacts.... we > > probably > dont > > have much to worry about :) > > > > when this happens id suggest : create a jira, folks will gladly > > help you to get your latest binaries published so that people can > > use the latest cascading bits from bigtop yum repos . > > > > just my thoughts, i assume this is correct though. > > > > > > On Mon, May 11, 2015 at 7:19 AM, Andre Kelpe > > <[email protected]> > > wrote: > > > > > Hi, > > > > > > the last couple of days I have been investigating the Cascading > > integration > > > for Bigtop. I have navigated the project and created RPM files for > > > two > of > > > our projects. Since we are outside of the ASF, things tend to be a > > > bit different, but I got it to work. I am currently doing the same > > > for the debian packages, which should not take too much time. > > > > > > Now to my actual questions: Bigtop supports a number of > > > distributions > and > > > has CI infrastructure to build the packages. I am wondering how we > > > as > the > > > Cascading community can work with that infrastructure for better > > > integration, or if we have to replicate all of it. Let me know, > > > what > > could > > > work best. I'd love to test and build all packages in an easy way, > > > but going through JIRA with a commiter reviewing and committing it > > > looks > > like a > > > pretty heavy process. OTOH if I replicate the infrastructure I > > > will end > > up > > > having to catch up with all infrastructure changes all the time. > > > > > > I am unsure what your way of maintaining things is, once a Bigtop > release > > > is made. I guess you are not targeting a rolling release, but > > > rather a stable release to be used for a longer time. How do > > > bugfix releases for packages go in? Suppose we have a Cascading > > > tool version x.y in Bigtop > > a.b. > > > and we make a critical bugfix release x.z, which we want to > > > distribute > to > > > Bigtop users as well. Would that then require a complete Bigtop > > > release with voting and all that or do you envision a faster track > > > like linux distros, which update packages for a given release. > > > > > > Thanks for your answers! > > > > > > - André > > > > > > -- > > > André Kelpe > > > [email protected] > > > http://concurrentinc.com > > > > > > > > > > > -- > > jay vyas > > > > > > -- > André Kelpe > [email protected] > http://concurrentinc.com >
