If we did want to do the below, the next consideration is how we would differentiate between artifacts produced by one nightly build versus another. Let's say for example we update Hadoop. All components that pull in Hadoop as a dependency will produce changed binary artifacts at the next build. These need to be versioned differently from earlier built packages. I think we would want to introduce a global build identifier (could be managed by our CI master), increment the identifier for every nightly, and incorporate it as a package version suffix.
On Thu, May 14, 2015 at 4:32 PM, Andrew Purtell <[email protected]> wrote: > The Apache notion of release is wholly scoped to source distributions. The > Bigtop source is the build scripting, package files, Puppet files, etc. > Generated binary packages are not included. > > I think we can be limber here by providing nightly -SNAPSHOT versions of > the Bigtop source on every day our git tree is updated. As part of the > process of pushing out the Bigtop source snapshot we can use CI to kick off > binary package builds, then place the resulting convenience binary packages > from the snapshot into similarly convenient apt or yum repositories. > Therefore on day N, there would be for example Cascading 1.2.3 binary > convenience artifacts in the convenience repositories. After check in of > Cascading package file updates (and related changes), on day N+1 there > would be Cascading 1.2.3 and 1.2.4 binary convenience artifacts in those > repos. > > Periodically we can plant a flag and promote a -SNAPSHOT into the next > official release. > > Then, rinse, and repeat. > > > On Tue, May 12, 2015 at 11:26 PM, <[email protected]> wrote: > >> Idea would like to work through with the group is post 1.0, evolving the >> CI and project process to be able to do real-time, or just-in-time builds. >> >> We could have a "bigtop core", which would be components like >> Hadoop/hdfs/zookeeper/etc. Thinks like hbase, hive, etc would be next >> level components, then things like pig, cascading, etc would be the real >> fast movers. >> >> Think another area we can investigate is components that want to >> integrate with bigtop but are not dependent on jvm stack and just the >> interfaces/api's of the core components. Seems those as well could be fast >> movers >> >> Don’t know if that answers your question or not Andre, hopefully with >> your guys help we can get some Cascading action going and get the >> build/release process evolving so no matter how fast you guys are releasing >> users can deploy the bits integrated fashion to other Bigtop components >> >> -----Original Message----- >> From: Evans Ye [mailto:[email protected]] >> Sent: Tuesday, May 12, 2015 9:16 AM >> To: [email protected] >> Subject: Re: ci infrastructure and release process for Cascading >> integration >> >> Hi Andre >> >> I take you're not asking technical questions but rather than wondering >> something related to the process or policy. >> >> AFAIK, at this moment we don't have a process to have the up-to-date >> packages available. >> (The Bigtop CI used to be working, but collapsed when most of the build >> slaves went out of available) >> >> So in your example, when upgrading upstream component version from 1.1 to >> 2.0, they would need to submit a patch and build the package by themselves. >> >> But, thanks to the EMR team sponsoring us EC2 resources, we're now >> rebuilding our CI infra, which surely would have nightly build(RPM/DEB) >> available. >> >> OTOH, I can recall that Nate is also working on bigtop repo nightly >> build, when that comes online we can always have SNAPSHOT repos available >> as well. >> >> Evans >> >> 2015-05-12 0:27 GMT+08:00 Andre Kelpe <[email protected]>: >> >> > If that is true, what would happen in the case where we want to make a >> > new major release available? Suppose we have project in 1.x and now as >> > a upstream we release 2.0. Would such a change still be covered by >> > this? How would somebody trying to rebuild all packages handle these >> > changes? If we submit a package update, the bigtop source release will >> > only have the spec files/debian directory from the time that a bigtop >> > release was made. Does that still work with the apache rules? >> > >> > - André >> > >> > On Mon, May 11, 2015 at 3:52 PM, jay vyas >> > <[email protected]> >> > wrote: >> > >> > > i think -- that -- since bigtop RPM/DEB packages ARE NOT bound by >> > > the >> > ASF >> > > guidelines, they are just binary convenience artifacts.... we >> > > probably >> > dont >> > > have much to worry about :) >> > > >> > > when this happens id suggest : create a jira, folks will gladly >> > > help you to get your latest binaries published so that people can >> > > use the latest cascading bits from bigtop yum repos . >> > > >> > > just my thoughts, i assume this is correct though. >> > > >> > > >> > > On Mon, May 11, 2015 at 7:19 AM, Andre Kelpe >> > > <[email protected]> >> > > wrote: >> > > >> > > > Hi, >> > > > >> > > > the last couple of days I have been investigating the Cascading >> > > integration >> > > > for Bigtop. I have navigated the project and created RPM files for >> > > > two >> > of >> > > > our projects. Since we are outside of the ASF, things tend to be a >> > > > bit different, but I got it to work. I am currently doing the same >> > > > for the debian packages, which should not take too much time. >> > > > >> > > > Now to my actual questions: Bigtop supports a number of >> > > > distributions >> > and >> > > > has CI infrastructure to build the packages. I am wondering how we >> > > > as >> > the >> > > > Cascading community can work with that infrastructure for better >> > > > integration, or if we have to replicate all of it. Let me know, >> > > > what >> > > could >> > > > work best. I'd love to test and build all packages in an easy way, >> > > > but going through JIRA with a commiter reviewing and committing it >> > > > looks >> > > like a >> > > > pretty heavy process. OTOH if I replicate the infrastructure I >> > > > will end >> > > up >> > > > having to catch up with all infrastructure changes all the time. >> > > > >> > > > I am unsure what your way of maintaining things is, once a Bigtop >> > release >> > > > is made. I guess you are not targeting a rolling release, but >> > > > rather a stable release to be used for a longer time. How do >> > > > bugfix releases for packages go in? Suppose we have a Cascading >> > > > tool version x.y in Bigtop >> > > a.b. >> > > > and we make a critical bugfix release x.z, which we want to >> > > > distribute >> > to >> > > > Bigtop users as well. Would that then require a complete Bigtop >> > > > release with voting and all that or do you envision a faster track >> > > > like linux distros, which update packages for a given release. >> > > > >> > > > Thanks for your answers! >> > > > >> > > > - André >> > > > >> > > > -- >> > > > André Kelpe >> > > > [email protected] >> > > > http://concurrentinc.com >> > > > >> > > >> > > >> > > >> > > -- >> > > jay vyas >> > > >> > >> > >> > >> > -- >> > André Kelpe >> > [email protected] >> > http://concurrentinc.com >> > >> >> > > > -- > Best regards, > > - Andy > > Problems worthy of attack prove their worth by hitting back. - Piet Hein > (via Tom White) > -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
