On Wed, Jun 21, 2017 at 10:53PM, Olaf Flebbe wrote: > Andrew, > > sorry to be sounding dismissive.
For it worth - I don't think you were dismissive. But whatever... I actually agree with your point: tarballs could be easily derived from the packages. And if anyone needs them we can have a target in the build system (please submit the patch), but I would be really against having them as a part of the standard artifacts' set. > rpm can be converted to cpio payload. deb is an ar archive containing > data.tar,gz with the files as payload. > > Details can be looked up. > > Nevertheless, please do not call "tar" an "improvement" with respect to > deployment compared to "rpm" or "deb". I see the usecase to extract just the > content without all the dependency management, service management, > verification and so on. Please just use rpm2cpio or ar. +1 Cos > > Am 21.06.2017 um 22:08 schrieb Andrew Purtell <[email protected]>: > > > >> you surely are making jokes when you are saying TAR is an improvement > > with respect to RPM/DEB.You surely know that you can unpack every RPM > > straight to the filesystem (DEB requires two steps), in case you'll like to. > > > > Not a joke and the condescension isn't helpful either. > > > > > > On Wed, Jun 21, 2017 at 10:02 AM, Olaf Flebbe <[email protected]> wrote: > > > >> Hi Andrew, > >> > >> you surely are making jokes when you are saying TAR is an improvement with > >> respect to RPM/DEB.You surely know that you can unpack every RPM straight > >> to the filesystem (DEB requires two steps), in case you'll like to. > >> > >> You surely know that one can easily host a complete docker based hadoop > >> cluster on a developer machine in the current git of bigtop. And that > >> docker toolbox, docker engine, docker for mac integrates really well with > >> Windows, Linux and MacOSX, working right out of the box (at least on MacOSX > >> and Linux) as it is right now within bigtop without manually tweaking > >> config files. > >> > >> I see no point in reproducing hive, hbase, ... hadoop tests -- most of > >> them single machine, fake cluster environment -- when we can have the real > >> thing, a cluster where we use docker for isolating nodes. When the tests do > >> not really work portable, that's a problem of other projects, not ours. > >> Let's fix it there. > >> > >> IMHO, if we could orchestrate k8s (kubernetes) (or docker-swarm, my > >> favorite) we could even chose to use a single host with some docker > >> instances or scale out to a cloud environment and have a reproducable > >> system without tweaking files. Of course there is much work to do to port > >> tests to the cloud environment, but these would be a tremendous value > >> added. > >> > >> Olaf > >> > >> > >> > >> > >>> Am 20.06.2017 um 23:12 schrieb Andrew Purtell <[email protected] > >>> : > >>> > >>> Yeah, we can build from git repos. Instead of archive URL you can > >> specify for each component a repo and reference by git-URL and branch, tag, > >> or SHA. > >>> > >>> Regarding tarball build targets, I was thinking of it as a packaging > >> improvement, an additional packaging target. It could make integration > >> testing more convenient too if you are not using containers or bare metal > >> systems where you own the whole filesystem. > >>> > >>>> On Jun 19, 2017, at 6:13 AM, Evans Ye <[email protected]> wrote: > >>>> > >>>> Hi Andy, > >>>> > >>>> Is it easier to have multiple tarballs to setup a cluster for > >> integration > >>>> tests? > >>>> I'm not on the Hadoop/HBase developer side so I have zero context. I was > >>>> just assuming that deploying a cluster for integration tests would be a > >>>> beneficial feature for them. > >>>> > >>>> Bringing up my discussion with Hadoop and HBase guys at Cloudera, them > >>>> mentioned two things specifically for Bigtop: > >>>> > >>>> a). build from git (which I think you've contributed that in Bigtop > >> already) > >>>> b). easy to run integration test framework > >>>> > >>>> I'm happy to have b). because either way we need to have it in our CI. > >>>> > >>>> > >>>> 2017-06-19 5:04 GMT+08:00 Andrew Purtell <[email protected]>: > >>>> > >>>>> IMHO, the easiest and fastest way to get the distribution aspect to be > >> more > >>>>> useful to more folks is to add a build target that generates plain > >> tarballs > >>>>> instead of distro-specific Linux packaging. People like us can take the > >>>>> tarballs and unpack them to environments where for various reasons we > >> don't > >>>>> want to do RPM management. Vendors like Cloudera can convert tarballs > >> to > >>>>> parcels, or whatever proprietary format is desired. > >>>>> > >>>>> > >>>>> > >>>>>> On Sun, Jun 18, 2017 at 12:13 PM, Evans Ye <[email protected]> > >> wrote: > >>>>>> > >>>>>> Hi folks, > >>>>>> > >>>>>> Many things happened during DataWorks Summit San Jose 2017. Some of > >> the > >>>>>> folks(Cos, Roman, Amir, Nate, Mike, etc) gathered together to discuss > >>>>> 1.2.1 > >>>>>> and the future 1.3 release of Bigtop. I'd like to get back those > >>>>>> discussions to the mailing list so that who can't make it there can > >> still > >>>>>> be with us for further discussions: > >>>>>> > >>>>>> * 1.2.1 release > >>>>>> a). Some of the folks expecting Docker on YARN to be back ported to > >> 2.7.4 > >>>>>> and included in the release > >>>>>> b). Get rotted code out of our code base: packaging, deployment, > >> testing, > >>>>>> etc > >>>>>> c). Get integration test to work in CI > >>>>>> > >>>>>> * 1.3.0 release > >>>>>> a). More machine learning integrations > >>>>>> b). K8S integration will be an interesting topic > >>>>>> > >>>>>> Please help me to complete the list if I miss something. :) > >>>>>> > >>>>>> > >>>>>> OTOH, for me specifically, I visited Cloudera for doing a tech talk. I > >>>>> meet > >>>>>> Sean Mackrory and there Hadoop and HBase lead. The pain point they're > >>>>>> having for a long time is not having an integration test framework for > >>>>>> there work on the bleeding edge. For example, whether a specific patch > >>>>> from > >>>>>> Hadoop breaks HBase or Hive? > >>>>>> > >>>>>> My thinking towards this is this is what Bigtop tries to solve at the > >>>>> very > >>>>>> beginning. We supposed to have folks from multiple projects to work > >> with > >>>>> us > >>>>>> to upgrade packages, and use our frameworks to properly integrate, > >> test > >>>>>> their code with other components. > >>>>>> > >>>>>> So, the future of Bigtop. I think tightly work with the other > >> communities > >>>>>> is a better way we move forward. But, that means something need to be > >>>>>> changed. For example, our distribution is somehow, from developers > >>>>>> perspective, old. Which can not support the integration and testing on > >>>>> the > >>>>>> bleeding edge. If we still like to release something suggested for > >>>>>> Production only, one of the solution is to have both dev and stable > >>>>>> releases in Bigtop, so developers can work on the dev branch and test > >>>>>> against newest components. In that case, people from other communities > >>>>>> might be possible to help us upgrade the package to the newer version, > >>>>>> which makes things easier. > >>>>>> > >>>>>> What do you guys think? Please join me for the discussion. > >>>>>> > >>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> Best regards, > >>>>> > >>>>> - Andy > >>>>> > >>>>> If you are given a choice, you believe you have acted freely. - Raymond > >>>>> Teller (via Peter Watts) > >>>>> > >> > >> > > > > > > -- > > Best regards, > > Andrew > > > > Words like orphans lost among the crosstalk, meaning torn from truth's > > decrepit hands > > - A23, Crosstalk >
signature.asc
Description: Digital signature
