I have been thinking about this for a time. Maybe tar.gz is actually
something we can do. Considering that the tech trend is container, k8s, and
cloud, having something easy and not over manufactured for downstream to
consume and integrate is an advantage of tar.gz. For us we keep focusing on
integration test and doing open source distribution of hadoop. The binary
format can be RPM/DEB, but not limited to.

Some triggers pushing me to this end:
1. People are using docker for packaging and deployment instead of rpm/deb.
2. Hadoop 3's messy layout don't fit into rpm/deb conventions. Maybe it's
because they think it's the right design for the container era.

Going down to the implementation level, we can do this by branching out
after do component builds. However all the integration tests will be broken
at the deployment stage. That means we need an entire new deployment tool.
Or we can leverage other existing solutions like Ambari?

BTW, I'm also curious how Ambari(hortonworks) integrates Hadoop 3 and
do.dependency, daemons...



Konstantin Boudnik <[email protected]> 於 2019年7月20日 週六 01:17 寫道:

> +1 on both Olaf's and Evans' points. Tarballs are messy stuff and are hard
> to
> control in operations: I believe the main reason for the existence of them
> in
> the official component releases is the simplicity of the media and lesser
> pressure on the community to integrate the software into a target stack (to
> Olaf's point).
>
> To the ARM's folks issue: aren't they able to use structured packages for
> some
> reason? Funny, this conversation happens every 2-3 years, like a clock ;)
>
> Thanks,
>   Cos
>
> On Sat, Jul 20, 2019 at 01:06AM, Evans Ye wrote:
> > From technical perspective Olaf's points are reasonable. However I'd like
> > to bridge the gap here so that we can strike a balance of what arm folks
> > need.
> >
> > Guodong let's back the story with some technical discussions. For
> example,
> > is it because your customer already implemented their deployment tool
> with
> > tar.gz? If so that's a strong reason from their perspective.
> > To me tar.gz has no much pros for production. For Apache projects, all
> the
> > release are source code hence tar.gz is the most simple way. Furthermore,
> > most of the projects will provide compiled binary tarball just for
> > convenient.
> >
> > Olaf Flebbe <[email protected]> 於 2019年7月19日 週五 上午3:40寫道:
> >
> > > Hi
> > >
> > > Maybe we have different target groups: For educational users, trying to
> > > figure out how everything works together, the single computer cluster
> is
> > > great.
> > >
> > > This Tar.gz artefact were never ever been sufficient to run in
> production.
> > > They did not contain 64bit libs for instance. They do not provide start
> > > scripts, error prone when you accidently using the wrong java, do not
> place
> > > logfiles in suitable places and so on.
> > >
> > > This tar artefacts and instructions are POC quality, that's it.
> > >
> > > Please be aware that Bigtop changes directory layout, directory
> > > permissions, configuration and startup of the original code in order to
> > > support large scale installations and automation.
> > >
> > > We have a much better alternative: Distribution integrated
> repositories.
> > > Youll only have to apt/yum install hadoop-hdfs-datanode and everything
> is
> > > already setup, including java, runscripts, directory layout and users.
> Then
> > > you can concentrate on configuring that beast. You do not even need no
> > > special instruction, since it is distribution native -- well, almost.
> If
> > > you feel that bigtops runscripts / layout is not suitable , you are
> very
> > > welcome to contribute.
> > >
> > > If i look at the instructions [1] . they contain tons of settings you
> > > should do. With our packages you can actually do exactly this, without
> > > bothering to install and configure all the dependencies.
> > >
> > > Olaf
> > >
> > >
> > >
> > >
> > >
> > > Von meinem iPad gesendet
> > >
> > > > Am 18.07.2019 um 08:57 schrieb Guodong Xu <[email protected]>:
> > > >
> > > > Hi, Evans
> > > >
> > > > Comments in below.
> > > >
> > > >>      On Tue, Jul 16, 2019 at 10:46 PM Evans Ye <[email protected]>
> > > wrote:
> > > >>
> > > >> I'm not objecting this, but I'd love to have more discussion to
> figure
> > > out
> > > >> whether this is the right thing to do. What I get from your
> proposal is
> > > >> users want to do things which RPM/DEB don't do well while tar.gz is
> > > good at
> > > >> it. However is it able to do it in another way which is far more
> > > beneficial
> > > >> in more scenarios? In general, it's like when users are asking for a
> > > faster
> > > >> horses, can we come up with cars?
> > > >>
> > > >> How about we start with the first step which is to elaborate why
> users
> > > >> choose to go for tar.gz?
> > > >
> > > >
> > > > Right, agree. Here is what I learned from users of Arm servers:
> > > >
> > > > One background for this is, currently, CDH and Hortonworks both have
> no
> > > > official release for Arm server yet. So, Bigtop is the only
> available and
> > > > verified distribution to them. As you know, with effort from the
> > > community
> > > > and Arm Inc., Linaro, Bigtop now has officially supported Arm64.
> > > >
> > > > To these users, before they start to use Bigtop on Arm, they are
> already
> > > > familiar with each component's individual installation and usage on
> x86.
> > > > Most of them are released in .tar.gz format. (i.e. Most apache big
> data
> > > > component doesn't release in deb/rpm. So, if we tell users that the
> only
> > > > available format in Bigtop is deb/rpm, this just hesitates them).
> > > >
> > > > Eg. For Hadoop, the official site for installation is here [1] and
> here
> > > > [2], their release format is .tar.gz.
> > > >
> > > > So,
> > > > 1. using .tar.gz is the minimum effort route for them to start their
> > > touch
> > > > with Bigtop (if we can support .tar.gz).
> > > > 2. Bigtop has all components tested. That provides very big
> confidence to
> > > > the users. So they like to use the binary built from Bigtop on Arm64
> > > > servers.
> > > >
> > > > [1]
> > > >
> > >
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html
> > > > [2] https://www.apache.org/dyn/closer.cgi/hadoop/common/
> > > >
> > > > -Guodong
> > > >
> > > >
> > > >>
> > > >> Guodong Xu <[email protected]> 於 2019年7月16日 週二 下午12:24寫道:
> > > >>
> > > >>> Hi, all
> > > >>>
> > > >>> This is a request for comments for a potential new feature. Please
> > > have a
> > > >>> look and let me know whether this adds value to the community.
> > > Appreciate
> > > >>> your opinions on this.
> > > >>>
> > > >>> As most of you know already, so far, Bigtop provides two packaging
> > > >> format:
> > > >>> deb and rpm. Users are recommended to install through apt and yum
> for
> > > >> their
> > > >>> respective linux distributions.
> > > >>>
> > > >>> Yet, there are still users coming to me and asking for the more
> > > >> traditional
> > > >>> .tar.gz packages. I know that shouldn't be a problem for x86 users,
> > > since
> > > >>> they can always download .tar.gz bin releases from each component's
> > > >>> official release website. But for users from other architectures,
> like
> > > >>> Arm64 and powerpc, story is different. .tar.gz are not readily
> > > available
> > > >> to
> > > >>> these architectures.
> > > >>>
> > > >>> In Bigtop, we are doing the job of building/packaging/testing big
> data
> > > >>> components in one single place. So, it makes sense that we step
> in, and
> > > >>> during our process of building each component, we keep a binary
> .tar.gz
> > > >>> copy for each component we supported.
> > > >>>
> > > >>> If you are supportive to the above idea, then next step will be
> how to
> > > >>> achieve that. I did some research actually. So, my method is to do
> two
> > > >>> things:
> > > >>> 0. this can be a subsequent task of either $target-deb or
> $target-rpm.
> > > It
> > > >>> depends on one of them two.
> > > >>> 1. add tar and cp into each component's 'do-component-build' file.
> > > >>> 2. add a new set of packaging tasks for each component. This means
> to
> > > >> add a
> > > >>> new task in genTasks() in packages.gradle.
> > > >>>  Something like
> > > >>> Task t = task "${target}-bin-tar" (
> > > >>> description: "Building .tar.gz binary for $target artifacts",
> > > >>> group: PACKAGES_GROUP) doLast { ...
> > > >>>
> > > >>>
> > > >>>
> > > >>> PS:
> > > >>> An example in 'do-component-build' for hadoop. Other components
> > > similar.
> > > >>>
> > > >>> diff --git a/bigtop-packages/src/common/hadoop/do-component-build
> > > >>> b/bigtop-packages/src/common/hadoop/do-component-build
> > > >>> index 2a1a6345..cb8ddd9d 100644
> > > >>> --- a/bigtop-packages/src/common/hadoop/do-component-build
> > > >>> +++ b/bigtop-packages/src/common/hadoop/do-component-build
> > > >>> @@ -146,3 +146,6 @@ cp -r target/staging/hadoop-project
> build/share/doc
> > > >>>
> > > >>> # Copy fuse output to the build directory
> > > >>> cp
> > > >>>
> > > >>>
> > > >>
> > >
> hadoop-hdfs-project/hadoop-hdfs-native-client/target/main/native/fuse-dfs/fuse_dfs
> > > >>> build/bin
> > > >>> +
> > > >>> +# Copy binary build result
> > > >>> +cp hadoop-dist/target/hadoop-${HADOOP_VERSION}.tar.gz ..
> > > >>>
> > > >>>
> > > >>> Best regards,
> > > >>> -Guodong
> > > >>>
> > > >>
> > >
>

Reply via email to