Yup, I saw that before. (Nice looking BTW) It's an awesome video for someone who'd like to know the detail about bigtop provisioner(the first half) as well as the packaging stuff(the second half).
How about we just put the video on bigtop wiki so that we can enrich the content? 2015-05-01 21:41 GMT+08:00 jay vyas <[email protected]>: > By the way, I did a hack session w/ the guys at BU on how to build/test > rpms using our existing vagrant workflow > > https://www.youtube.com/watch?v=4GfcKEjO6e8 > > You can get a good feel for some of the lower level details hard to capture > in text by watching it. > > On Thu, Apr 30, 2015 at 6:30 PM, Konstantin Boudnik <[email protected]> > wrote: > > > Andrew, > > > > do you might putting this on our wiki? Such a great and well-put > > explanation! > > I am sure it will help a lot of new contributors to get up to speed much > > quicker! > > > > Thanks! > > Cos > > > > On Tue, Apr 28, 2015 at 04:01PM, Andrew Purtell wrote: > > > Inline > > > > > > On Tue, Apr 28, 2015 at 8:51 AM, Andre Kelpe <[email protected] > > > > > wrote: > > > > > > > Hi, > > > > > > > > I am currently learning the ins and outs of bigtop to work on the > > Cascading > > > > integration (https://issues.apache.org/jira/browse/BIGTOP-1766). I > > have a > > > > few questions around packaging in bigtop: > > > > > > > > 1) most linux distros have packaging guidelines that should be > > followed. > > > > Does bigtop follow any set of rules in particular? Is there a linting > > tool > > > > for spec files etc? > > > > > > > > > > This is distro specific. RedHat family distributions (RHEL, Fedora, > > Centos, > > > Amazon Linux) offer 'rpmlint'. You can install it and run it by hand. > > From > > > personal experience if you build deb packages on Ubuntu the package > build > > > will run the lintian tool automatically. > > > > > > > > > > 2) Related to 1): Does bigtop require to follow a certain directory > > layout? > > > > Our tools are currently meant to be untarred and used as is, if > bigtop > > > > requires them to be split over the file-system, we will have to work > on > > > > that upstream before they can be included. > > > > > > > > > > Yes, broadly speaking we follow the Linux standard base (LSB). A > typical > > > package build happens in four steps. We move files around in the third > > step > > > to make packages look more like LSB. Let me take you through one > package > > as > > > an example: > > > > > > Step 1. Download source tarball from the software release site and > expand > > > it. > > > > > > Step 2. do-package-build > > > > > > Here, for example, see what we do for ZooKeeper: > > > > > > https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/zookeeper/do-component-build > > > . We kick off a build of the component's binary artifacts while first > > > normalizing dependency versions according to the release BOM. > > > > > > Step 3. install_<component>.sh > > > > > > Again let's look at the ZK package: > > > > > > https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/zookeeper/install_zookeeper.sh > > > . Here we take the resulting tarball from the component build, expand > it, > > > and move the locations of various types of files around to be more > > > LSB-like. > > > > > > Step 4. Native packager > > > > > > Finally we hand off the expanded and munged result from step 3 to the > > > native packager. For ZK, the RPM specfile used is here: > > > > > > https://github.com/apache/bigtop/blob/master/bigtop-packages/src/rpm/zookeeper/SPECS/zookeeper.spec > > > . The Debian package control files are here: > > > > > > https://github.com/apache/bigtop/tree/master/bigtop-packages/src/deb/zookeeper > > > > > > > > > > > > > > > > 3) I noticed that the packages are build from source instead of > > re-using > > > > binary releases. Is that a strict requirement or does it just happen > > to be > > > > that way? For the Cascading integration I was planning on downloading > > our > > > > binary releases so that bigtop ship with the same bits as our SDK. > > > > > > > > > > We typically build packages from source so we can normalize > > dependencies. > > > For example, if a given Bigtop release ships with Hadoop 2.6.0 but the > > > Cascading SDK includes 2.5.1 artifacts, this would be ugly at best and > > > broken at worst. > > > > > > > > > > 4) What is your take on packaging standalone libraries? I noticed > that > > most > > > > parts of bigtop are tools in the broader sense. Something one can > > invoke on > > > > the command line, but there is also a package for apache crunch, > which > > is a > > > > library. What is the reasoning here? Would it make sense to build > > packages > > > > for libraries in the Cascading eco-system? > > > > > > > > > > > I'm not sure we have anything that amounts to a policy here. Crunch > isn't > > > the only case. We package the DataFu library of UDFs for Pig. We > package > > > the Phoenix SQL skin add-on for HBase. We also package Tez, which is a > > YARN > > > application requiring Hadoop, and although it could be useful on its > own > > > it's meant to be picked up and used by the Hive and Pig packages. > > > > > > If a champion for a component shows up we will give it a look. We could > > > absolutely build a core Cascading package and then a number of library > or > > > add-on packages, if that's how you would like to set things up as > > champion > > > or maintainer of same. > > > > > > > > > > > > > Thanks for your answers! > > > > > > > > - André > > > > > > > > -- > > > > André Kelpe > > > > [email protected] > > > > http://concurrentinc.com > > > > > > > > > > > > > > > > -- > > > Best regards, > > > > > > - Andy > > > > > > Problems worthy of attack prove their worth by hitting back. - Piet > Hein > > > (via Tom White) > > > > > > -- > jay vyas >
