We also talked briefly about the idea of post 1.0 maybe leveraging semver style where 1.0.1, or 1.1 might be release, and the 1.0.1 could be patch for a component, where maybe a 1.1.0 would be a rev where for instance cascading was revved but the "bigtop core" was the same
-----Original Message----- From: Konstantin Boudnik [mailto:[email protected]] Sent: Thursday, May 14, 2015 4:49 PM To: [email protected] Subject: Re: ci infrastructure and release process for Cascading integration All packages include the build stamp that increases monotonically: that guarantees the name uniqueness and ability to tell them apart for update purposes and such. Cos On Thu, May 14, 2015 at 04:41PM, Andrew Purtell wrote: > If we did want to do the below, the next consideration is how we would > differentiate between artifacts produced by one nightly build versus > another. Let's say for example we update Hadoop. All components that > pull in Hadoop as a dependency will produce changed binary artifacts > at the next build. These need to be versioned differently from earlier built > packages. > I think we would want to introduce a global build identifier (could be > managed by our CI master), increment the identifier for every nightly, > and incorporate it as a package version suffix. > > > On Thu, May 14, 2015 at 4:32 PM, Andrew Purtell <[email protected]> wrote: > > > The Apache notion of release is wholly scoped to source > > distributions. The Bigtop source is the build scripting, package files, > > Puppet files, etc. > > Generated binary packages are not included. > > > > I think we can be limber here by providing nightly -SNAPSHOT > > versions of the Bigtop source on every day our git tree is updated. > > As part of the process of pushing out the Bigtop source snapshot we > > can use CI to kick off binary package builds, then place the > > resulting convenience binary packages from the snapshot into similarly > > convenient apt or yum repositories. > > Therefore on day N, there would be for example Cascading 1.2.3 > > binary convenience artifacts in the convenience repositories. After > > check in of Cascading package file updates (and related changes), on > > day N+1 there would be Cascading 1.2.3 and 1.2.4 binary convenience > > artifacts in those repos. > > > > Periodically we can plant a flag and promote a -SNAPSHOT into the > > next official release. > > > > Then, rinse, and repeat. > > > > > > On Tue, May 12, 2015 at 11:26 PM, <[email protected]> wrote: > > > >> Idea would like to work through with the group is post 1.0, > >> evolving the CI and project process to be able to do real-time, or > >> just-in-time builds. > >> > >> We could have a "bigtop core", which would be components like > >> Hadoop/hdfs/zookeeper/etc. Thinks like hbase, hive, etc would be > >> next level components, then things like pig, cascading, etc would > >> be the real fast movers. > >> > >> Think another area we can investigate is components that want to > >> integrate with bigtop but are not dependent on jvm stack and just > >> the interfaces/api's of the core components. Seems those as well > >> could be fast movers > >> > >> Don’t know if that answers your question or not Andre, hopefully > >> with your guys help we can get some Cascading action going and get > >> the build/release process evolving so no matter how fast you guys > >> are releasing users can deploy the bits integrated fashion to other > >> Bigtop components > >> > >> -----Original Message----- > >> From: Evans Ye [mailto:[email protected]] > >> Sent: Tuesday, May 12, 2015 9:16 AM > >> To: [email protected] > >> Subject: Re: ci infrastructure and release process for Cascading > >> integration > >> > >> Hi Andre > >> > >> I take you're not asking technical questions but rather than > >> wondering something related to the process or policy. > >> > >> AFAIK, at this moment we don't have a process to have the > >> up-to-date packages available. > >> (The Bigtop CI used to be working, but collapsed when most of the > >> build slaves went out of available) > >> > >> So in your example, when upgrading upstream component version from > >> 1.1 to 2.0, they would need to submit a patch and build the package by > >> themselves. > >> > >> But, thanks to the EMR team sponsoring us EC2 resources, we're now > >> rebuilding our CI infra, which surely would have nightly > >> build(RPM/DEB) available. > >> > >> OTOH, I can recall that Nate is also working on bigtop repo nightly > >> build, when that comes online we can always have SNAPSHOT repos > >> available as well. > >> > >> Evans > >> > >> 2015-05-12 0:27 GMT+08:00 Andre Kelpe <[email protected]>: > >> > >> > If that is true, what would happen in the case where we want to > >> > make a new major release available? Suppose we have project in > >> > 1.x and now as a upstream we release 2.0. Would such a change > >> > still be covered by this? How would somebody trying to rebuild > >> > all packages handle these changes? If we submit a package update, > >> > the bigtop source release will only have the spec files/debian > >> > directory from the time that a bigtop release was made. Does that still > >> > work with the apache rules? > >> > > >> > - André > >> > > >> > On Mon, May 11, 2015 at 3:52 PM, jay vyas > >> > <[email protected]> > >> > wrote: > >> > > >> > > i think -- that -- since bigtop RPM/DEB packages ARE NOT bound > >> > > by the > >> > ASF > >> > > guidelines, they are just binary convenience artifacts.... we > >> > > probably > >> > dont > >> > > have much to worry about :) > >> > > > >> > > when this happens id suggest : create a jira, folks will > >> > > gladly help you to get your latest binaries published so that > >> > > people can use the latest cascading bits from bigtop yum repos . > >> > > > >> > > just my thoughts, i assume this is correct though. > >> > > > >> > > > >> > > On Mon, May 11, 2015 at 7:19 AM, Andre Kelpe > >> > > <[email protected]> > >> > > wrote: > >> > > > >> > > > Hi, > >> > > > > >> > > > the last couple of days I have been investigating the > >> > > > Cascading > >> > > integration > >> > > > for Bigtop. I have navigated the project and created RPM > >> > > > files for two > >> > of > >> > > > our projects. Since we are outside of the ASF, things tend to > >> > > > be a bit different, but I got it to work. I am currently > >> > > > doing the same for the debian packages, which should not take too > >> > > > much time. > >> > > > > >> > > > Now to my actual questions: Bigtop supports a number of > >> > > > distributions > >> > and > >> > > > has CI infrastructure to build the packages. I am wondering > >> > > > how we as > >> > the > >> > > > Cascading community can work with that infrastructure for > >> > > > better integration, or if we have to replicate all of it. Let > >> > > > me know, what > >> > > could > >> > > > work best. I'd love to test and build all packages in an easy > >> > > > way, but going through JIRA with a commiter reviewing and > >> > > > committing it looks > >> > > like a > >> > > > pretty heavy process. OTOH if I replicate the infrastructure > >> > > > I will end > >> > > up > >> > > > having to catch up with all infrastructure changes all the time. > >> > > > > >> > > > I am unsure what your way of maintaining things is, once a > >> > > > Bigtop > >> > release > >> > > > is made. I guess you are not targeting a rolling release, but > >> > > > rather a stable release to be used for a longer time. How do > >> > > > bugfix releases for packages go in? Suppose we have a > >> > > > Cascading tool version x.y in Bigtop > >> > > a.b. > >> > > > and we make a critical bugfix release x.z, which we want to > >> > > > distribute > >> > to > >> > > > Bigtop users as well. Would that then require a complete > >> > > > Bigtop release with voting and all that or do you envision a > >> > > > faster track like linux distros, which update packages for a given > >> > > > release. > >> > > > > >> > > > Thanks for your answers! > >> > > > > >> > > > - André > >> > > > > >> > > > -- > >> > > > André Kelpe > >> > > > [email protected] > >> > > > http://concurrentinc.com > >> > > > > >> > > > >> > > > >> > > > >> > > -- > >> > > jay vyas > >> > > > >> > > >> > > >> > > >> > -- > >> > André Kelpe > >> > [email protected] > >> > http://concurrentinc.com > >> > > >> > >> > > > > > > -- > > Best regards, > > > > - Andy > > > > Problems worthy of attack prove their worth by hitting back. - Piet > > Hein (via Tom White) > > > > > > -- > Best regards, > > - Andy > > Problems worthy of attack prove their worth by hitting back. - Piet > Hein (via Tom White)
