Yeah, we can build from git repos. Instead of archive URL you can specify for each component a repo and reference by git-URL and branch, tag, or SHA.
Regarding tarball build targets, I was thinking of it as a packaging improvement, an additional packaging target. It could make integration testing more convenient too if you are not using containers or bare metal systems where you own the whole filesystem. > On Jun 19, 2017, at 6:13 AM, Evans Ye <[email protected]> wrote: > > Hi Andy, > > Is it easier to have multiple tarballs to setup a cluster for integration > tests? > I'm not on the Hadoop/HBase developer side so I have zero context. I was > just assuming that deploying a cluster for integration tests would be a > beneficial feature for them. > > Bringing up my discussion with Hadoop and HBase guys at Cloudera, them > mentioned two things specifically for Bigtop: > > a). build from git (which I think you've contributed that in Bigtop already) > b). easy to run integration test framework > > I'm happy to have b). because either way we need to have it in our CI. > > > 2017-06-19 5:04 GMT+08:00 Andrew Purtell <[email protected]>: > >> IMHO, the easiest and fastest way to get the distribution aspect to be more >> useful to more folks is to add a build target that generates plain tarballs >> instead of distro-specific Linux packaging. People like us can take the >> tarballs and unpack them to environments where for various reasons we don't >> want to do RPM management. Vendors like Cloudera can convert tarballs to >> parcels, or whatever proprietary format is desired. >> >> >> >>> On Sun, Jun 18, 2017 at 12:13 PM, Evans Ye <[email protected]> wrote: >>> >>> Hi folks, >>> >>> Many things happened during DataWorks Summit San Jose 2017. Some of the >>> folks(Cos, Roman, Amir, Nate, Mike, etc) gathered together to discuss >> 1.2.1 >>> and the future 1.3 release of Bigtop. I'd like to get back those >>> discussions to the mailing list so that who can't make it there can still >>> be with us for further discussions: >>> >>> * 1.2.1 release >>> a). Some of the folks expecting Docker on YARN to be back ported to 2.7.4 >>> and included in the release >>> b). Get rotted code out of our code base: packaging, deployment, testing, >>> etc >>> c). Get integration test to work in CI >>> >>> * 1.3.0 release >>> a). More machine learning integrations >>> b). K8S integration will be an interesting topic >>> >>> Please help me to complete the list if I miss something. :) >>> >>> >>> OTOH, for me specifically, I visited Cloudera for doing a tech talk. I >> meet >>> Sean Mackrory and there Hadoop and HBase lead. The pain point they're >>> having for a long time is not having an integration test framework for >>> there work on the bleeding edge. For example, whether a specific patch >> from >>> Hadoop breaks HBase or Hive? >>> >>> My thinking towards this is this is what Bigtop tries to solve at the >> very >>> beginning. We supposed to have folks from multiple projects to work with >> us >>> to upgrade packages, and use our frameworks to properly integrate, test >>> their code with other components. >>> >>> So, the future of Bigtop. I think tightly work with the other communities >>> is a better way we move forward. But, that means something need to be >>> changed. For example, our distribution is somehow, from developers >>> perspective, old. Which can not support the integration and testing on >> the >>> bleeding edge. If we still like to release something suggested for >>> Production only, one of the solution is to have both dev and stable >>> releases in Bigtop, so developers can work on the dev branch and test >>> against newest components. In that case, people from other communities >>> might be possible to help us upgrade the package to the newer version, >>> which makes things easier. >>> >>> What do you guys think? Please join me for the discussion. >>> >> >> >> >> -- >> Best regards, >> >> - Andy >> >> If you are given a choice, you believe you have acted freely. - Raymond >> Teller (via Peter Watts) >>
