All packages include the build stamp that increases monotonically: that guarantees the name uniqueness and ability to tell them apart for update purposes and such.
Cos On Thu, May 14, 2015 at 04:41PM, Andrew Purtell wrote: > If we did want to do the below, the next consideration is how we would > differentiate between artifacts produced by one nightly build versus > another. Let's say for example we update Hadoop. All components that pull > in Hadoop as a dependency will produce changed binary artifacts at the next > build. These need to be versioned differently from earlier built packages. > I think we would want to introduce a global build identifier (could be > managed by our CI master), increment the identifier for every nightly, and > incorporate it as a package version suffix. > > > On Thu, May 14, 2015 at 4:32 PM, Andrew Purtell <[email protected]> wrote: > > > The Apache notion of release is wholly scoped to source distributions. The > > Bigtop source is the build scripting, package files, Puppet files, etc. > > Generated binary packages are not included. > > > > I think we can be limber here by providing nightly -SNAPSHOT versions of > > the Bigtop source on every day our git tree is updated. As part of the > > process of pushing out the Bigtop source snapshot we can use CI to kick off > > binary package builds, then place the resulting convenience binary packages > > from the snapshot into similarly convenient apt or yum repositories. > > Therefore on day N, there would be for example Cascading 1.2.3 binary > > convenience artifacts in the convenience repositories. After check in of > > Cascading package file updates (and related changes), on day N+1 there > > would be Cascading 1.2.3 and 1.2.4 binary convenience artifacts in those > > repos. > > > > Periodically we can plant a flag and promote a -SNAPSHOT into the next > > official release. > > > > Then, rinse, and repeat. > > > > > > On Tue, May 12, 2015 at 11:26 PM, <[email protected]> wrote: > > > >> Idea would like to work through with the group is post 1.0, evolving the > >> CI and project process to be able to do real-time, or just-in-time builds. > >> > >> We could have a "bigtop core", which would be components like > >> Hadoop/hdfs/zookeeper/etc. Thinks like hbase, hive, etc would be next > >> level components, then things like pig, cascading, etc would be the real > >> fast movers. > >> > >> Think another area we can investigate is components that want to > >> integrate with bigtop but are not dependent on jvm stack and just the > >> interfaces/api's of the core components. Seems those as well could be fast > >> movers > >> > >> Don’t know if that answers your question or not Andre, hopefully with > >> your guys help we can get some Cascading action going and get the > >> build/release process evolving so no matter how fast you guys are releasing > >> users can deploy the bits integrated fashion to other Bigtop components > >> > >> -----Original Message----- > >> From: Evans Ye [mailto:[email protected]] > >> Sent: Tuesday, May 12, 2015 9:16 AM > >> To: [email protected] > >> Subject: Re: ci infrastructure and release process for Cascading > >> integration > >> > >> Hi Andre > >> > >> I take you're not asking technical questions but rather than wondering > >> something related to the process or policy. > >> > >> AFAIK, at this moment we don't have a process to have the up-to-date > >> packages available. > >> (The Bigtop CI used to be working, but collapsed when most of the build > >> slaves went out of available) > >> > >> So in your example, when upgrading upstream component version from 1.1 to > >> 2.0, they would need to submit a patch and build the package by themselves. > >> > >> But, thanks to the EMR team sponsoring us EC2 resources, we're now > >> rebuilding our CI infra, which surely would have nightly build(RPM/DEB) > >> available. > >> > >> OTOH, I can recall that Nate is also working on bigtop repo nightly > >> build, when that comes online we can always have SNAPSHOT repos available > >> as well. > >> > >> Evans > >> > >> 2015-05-12 0:27 GMT+08:00 Andre Kelpe <[email protected]>: > >> > >> > If that is true, what would happen in the case where we want to make a > >> > new major release available? Suppose we have project in 1.x and now as > >> > a upstream we release 2.0. Would such a change still be covered by > >> > this? How would somebody trying to rebuild all packages handle these > >> > changes? If we submit a package update, the bigtop source release will > >> > only have the spec files/debian directory from the time that a bigtop > >> > release was made. Does that still work with the apache rules? > >> > > >> > - André > >> > > >> > On Mon, May 11, 2015 at 3:52 PM, jay vyas > >> > <[email protected]> > >> > wrote: > >> > > >> > > i think -- that -- since bigtop RPM/DEB packages ARE NOT bound by > >> > > the > >> > ASF > >> > > guidelines, they are just binary convenience artifacts.... we > >> > > probably > >> > dont > >> > > have much to worry about :) > >> > > > >> > > when this happens id suggest : create a jira, folks will gladly > >> > > help you to get your latest binaries published so that people can > >> > > use the latest cascading bits from bigtop yum repos . > >> > > > >> > > just my thoughts, i assume this is correct though. > >> > > > >> > > > >> > > On Mon, May 11, 2015 at 7:19 AM, Andre Kelpe > >> > > <[email protected]> > >> > > wrote: > >> > > > >> > > > Hi, > >> > > > > >> > > > the last couple of days I have been investigating the Cascading > >> > > integration > >> > > > for Bigtop. I have navigated the project and created RPM files for > >> > > > two > >> > of > >> > > > our projects. Since we are outside of the ASF, things tend to be a > >> > > > bit different, but I got it to work. I am currently doing the same > >> > > > for the debian packages, which should not take too much time. > >> > > > > >> > > > Now to my actual questions: Bigtop supports a number of > >> > > > distributions > >> > and > >> > > > has CI infrastructure to build the packages. I am wondering how we > >> > > > as > >> > the > >> > > > Cascading community can work with that infrastructure for better > >> > > > integration, or if we have to replicate all of it. Let me know, > >> > > > what > >> > > could > >> > > > work best. I'd love to test and build all packages in an easy way, > >> > > > but going through JIRA with a commiter reviewing and committing it > >> > > > looks > >> > > like a > >> > > > pretty heavy process. OTOH if I replicate the infrastructure I > >> > > > will end > >> > > up > >> > > > having to catch up with all infrastructure changes all the time. > >> > > > > >> > > > I am unsure what your way of maintaining things is, once a Bigtop > >> > release > >> > > > is made. I guess you are not targeting a rolling release, but > >> > > > rather a stable release to be used for a longer time. How do > >> > > > bugfix releases for packages go in? Suppose we have a Cascading > >> > > > tool version x.y in Bigtop > >> > > a.b. > >> > > > and we make a critical bugfix release x.z, which we want to > >> > > > distribute > >> > to > >> > > > Bigtop users as well. Would that then require a complete Bigtop > >> > > > release with voting and all that or do you envision a faster track > >> > > > like linux distros, which update packages for a given release. > >> > > > > >> > > > Thanks for your answers! > >> > > > > >> > > > - André > >> > > > > >> > > > -- > >> > > > André Kelpe > >> > > > [email protected] > >> > > > http://concurrentinc.com > >> > > > > >> > > > >> > > > >> > > > >> > > -- > >> > > jay vyas > >> > > > >> > > >> > > >> > > >> > -- > >> > André Kelpe > >> > [email protected] > >> > http://concurrentinc.com > >> > > >> > >> > > > > > > -- > > Best regards, > > > > - Andy > > > > Problems worthy of attack prove their worth by hitting back. - Piet Hein > > (via Tom White) > > > > > > -- > Best regards, > > - Andy > > Problems worthy of attack prove their worth by hitting back. - Piet Hein > (via Tom White)
