Putting technical debate aside, it would be appreciated if we can have discussions peacefully. After all all of us are just trying to make bigtop a better software. :)
Andy, Can you elaborate how tarballs can make integration tests more easily? Here I'm with Olaf that doing real cluster integration tests for compatibilities, operational issues like rolling upgrade are something we should be more focused on. Jay, I'd like to count on your expertise on k8s. I know a little about Swarm, but zero about k8s. How do you imaging the fsimages and edit logs being stored on k8s? I assume there must be some distributed store for these files so that leveraging docker for dynamic deployment is possible. 2017-06-22 4:53 GMT+08:00 Olaf Flebbe <[email protected]>: > Andrew, > > sorry to be sounding dismissive. > > rpm can be converted to cpio payload. deb is an ar archive containing > data.tar,gz with the files as payload. > > Details can be looked up. > > Nevertheless, please do not call "tar" an "improvement" with respect to > deployment compared to "rpm" or "deb". I see the usecase to extract just > the content without all the dependency management, service management, > verification and so on. Please just use rpm2cpio or ar. > > Olaf > > > > Am 21.06.2017 um 22:08 schrieb Andrew Purtell <[email protected]>: > > > >> you surely are making jokes when you are saying TAR is an improvement > > with respect to RPM/DEB.You surely know that you can unpack every RPM > > straight to the filesystem (DEB requires two steps), in case you'll like > to. > > > > Not a joke and the condescension isn't helpful either. > > > > > > On Wed, Jun 21, 2017 at 10:02 AM, Olaf Flebbe <[email protected]> wrote: > > > >> Hi Andrew, > >> > >> you surely are making jokes when you are saying TAR is an improvement > with > >> respect to RPM/DEB.You surely know that you can unpack every RPM > straight > >> to the filesystem (DEB requires two steps), in case you'll like to. > >> > >> You surely know that one can easily host a complete docker based hadoop > >> cluster on a developer machine in the current git of bigtop. And that > >> docker toolbox, docker engine, docker for mac integrates really well > with > >> Windows, Linux and MacOSX, working right out of the box (at least on > MacOSX > >> and Linux) as it is right now within bigtop without manually tweaking > >> config files. > >> > >> I see no point in reproducing hive, hbase, ... hadoop tests -- most of > >> them single machine, fake cluster environment -- when we can have the > real > >> thing, a cluster where we use docker for isolating nodes. When the > tests do > >> not really work portable, that's a problem of other projects, not ours. > >> Let's fix it there. > >> > >> IMHO, if we could orchestrate k8s (kubernetes) (or docker-swarm, my > >> favorite) we could even chose to use a single host with some docker > >> instances or scale out to a cloud environment and have a reproducable > >> system without tweaking files. Of course there is much work to do to > port > >> tests to the cloud environment, but these would be a tremendous value > added. > >> > >> Olaf > >> > >> > >> > >> > >>> Am 20.06.2017 um 23:12 schrieb Andrew Purtell < > [email protected] > >>> : > >>> > >>> Yeah, we can build from git repos. Instead of archive URL you can > >> specify for each component a repo and reference by git-URL and branch, > tag, > >> or SHA. > >>> > >>> Regarding tarball build targets, I was thinking of it as a packaging > >> improvement, an additional packaging target. It could make integration > >> testing more convenient too if you are not using containers or bare > metal > >> systems where you own the whole filesystem. > >>> > >>>> On Jun 19, 2017, at 6:13 AM, Evans Ye <[email protected]> wrote: > >>>> > >>>> Hi Andy, > >>>> > >>>> Is it easier to have multiple tarballs to setup a cluster for > >> integration > >>>> tests? > >>>> I'm not on the Hadoop/HBase developer side so I have zero context. I > was > >>>> just assuming that deploying a cluster for integration tests would be > a > >>>> beneficial feature for them. > >>>> > >>>> Bringing up my discussion with Hadoop and HBase guys at Cloudera, them > >>>> mentioned two things specifically for Bigtop: > >>>> > >>>> a). build from git (which I think you've contributed that in Bigtop > >> already) > >>>> b). easy to run integration test framework > >>>> > >>>> I'm happy to have b). because either way we need to have it in our CI. > >>>> > >>>> > >>>> 2017-06-19 5:04 GMT+08:00 Andrew Purtell <[email protected]>: > >>>> > >>>>> IMHO, the easiest and fastest way to get the distribution aspect to > be > >> more > >>>>> useful to more folks is to add a build target that generates plain > >> tarballs > >>>>> instead of distro-specific Linux packaging. People like us can take > the > >>>>> tarballs and unpack them to environments where for various reasons we > >> don't > >>>>> want to do RPM management. Vendors like Cloudera can convert tarballs > >> to > >>>>> parcels, or whatever proprietary format is desired. > >>>>> > >>>>> > >>>>> > >>>>>> On Sun, Jun 18, 2017 at 12:13 PM, Evans Ye <[email protected]> > >> wrote: > >>>>>> > >>>>>> Hi folks, > >>>>>> > >>>>>> Many things happened during DataWorks Summit San Jose 2017. Some of > >> the > >>>>>> folks(Cos, Roman, Amir, Nate, Mike, etc) gathered together to > discuss > >>>>> 1.2.1 > >>>>>> and the future 1.3 release of Bigtop. I'd like to get back those > >>>>>> discussions to the mailing list so that who can't make it there can > >> still > >>>>>> be with us for further discussions: > >>>>>> > >>>>>> * 1.2.1 release > >>>>>> a). Some of the folks expecting Docker on YARN to be back ported to > >> 2.7.4 > >>>>>> and included in the release > >>>>>> b). Get rotted code out of our code base: packaging, deployment, > >> testing, > >>>>>> etc > >>>>>> c). Get integration test to work in CI > >>>>>> > >>>>>> * 1.3.0 release > >>>>>> a). More machine learning integrations > >>>>>> b). K8S integration will be an interesting topic > >>>>>> > >>>>>> Please help me to complete the list if I miss something. :) > >>>>>> > >>>>>> > >>>>>> OTOH, for me specifically, I visited Cloudera for doing a tech > talk. I > >>>>> meet > >>>>>> Sean Mackrory and there Hadoop and HBase lead. The pain point > they're > >>>>>> having for a long time is not having an integration test framework > for > >>>>>> there work on the bleeding edge. For example, whether a specific > patch > >>>>> from > >>>>>> Hadoop breaks HBase or Hive? > >>>>>> > >>>>>> My thinking towards this is this is what Bigtop tries to solve at > the > >>>>> very > >>>>>> beginning. We supposed to have folks from multiple projects to work > >> with > >>>>> us > >>>>>> to upgrade packages, and use our frameworks to properly integrate, > >> test > >>>>>> their code with other components. > >>>>>> > >>>>>> So, the future of Bigtop. I think tightly work with the other > >> communities > >>>>>> is a better way we move forward. But, that means something need to > be > >>>>>> changed. For example, our distribution is somehow, from developers > >>>>>> perspective, old. Which can not support the integration and testing > on > >>>>> the > >>>>>> bleeding edge. If we still like to release something suggested for > >>>>>> Production only, one of the solution is to have both dev and stable > >>>>>> releases in Bigtop, so developers can work on the dev branch and > test > >>>>>> against newest components. In that case, people from other > communities > >>>>>> might be possible to help us upgrade the package to the newer > version, > >>>>>> which makes things easier. > >>>>>> > >>>>>> What do you guys think? Please join me for the discussion. > >>>>>> > >>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> Best regards, > >>>>> > >>>>> - Andy > >>>>> > >>>>> If you are given a choice, you believe you have acted freely. - > Raymond > >>>>> Teller (via Peter Watts) > >>>>> > >> > >> > > > > > > -- > > Best regards, > > Andrew > > > > Words like orphans lost among the crosstalk, meaning torn from truth's > > decrepit hands > > - A23, Crosstalk > >
