Support Evans thinking. Especially if Linaro can contribute towards getting
.tar.gz working, would we get a 'go ahead'  ?

On Sat, Jul 27, 2019 at 1:02 PM Evans Ye <[email protected]> wrote:

> I have been thinking about this for a time. Maybe tar.gz is actually
> something we can do. Considering that the tech trend is container, k8s, and
> cloud, having something easy and not over manufactured for downstream to
> consume and integrate is an advantage of tar.gz. For us we keep focusing on
> integration test and doing open source distribution of hadoop. The binary
> format can be RPM/DEB, but not limited to.
>
> Some triggers pushing me to this end:
> 1. People are using docker for packaging and deployment instead of rpm/deb.
> 2. Hadoop 3's messy layout don't fit into rpm/deb conventions. Maybe it's
> because they think it's the right design for the container era.
>
> Going down to the implementation level, we can do this by branching out
> after do component builds. However all the integration tests will be broken
> at the deployment stage. That means we need an entire new deployment tool.
> Or we can leverage other existing solutions like Ambari?
>
> BTW, I'm also curious how Ambari(hortonworks) integrates Hadoop 3 and
> do.dependency, daemons...
>
>
>
> Konstantin Boudnik <[email protected]> 於 2019年7月20日 週六 01:17 寫道:
>
> > +1 on both Olaf's and Evans' points. Tarballs are messy stuff and are
> hard
> > to
> > control in operations: I believe the main reason for the existence of
> them
> > in
> > the official component releases is the simplicity of the media and lesser
> > pressure on the community to integrate the software into a target stack
> (to
> > Olaf's point).
> >
> > To the ARM's folks issue: aren't they able to use structured packages for
> > some
> > reason? Funny, this conversation happens every 2-3 years, like a clock ;)
> >
> > Thanks,
> >   Cos
> >
> > On Sat, Jul 20, 2019 at 01:06AM, Evans Ye wrote:
> > > From technical perspective Olaf's points are reasonable. However I'd
> like
> > > to bridge the gap here so that we can strike a balance of what arm
> folks
> > > need.
> > >
> > > Guodong let's back the story with some technical discussions. For
> > example,
> > > is it because your customer already implemented their deployment tool
> > with
> > > tar.gz? If so that's a strong reason from their perspective.
> > > To me tar.gz has no much pros for production. For Apache projects, all
> > the
> > > release are source code hence tar.gz is the most simple way.
> Furthermore,
> > > most of the projects will provide compiled binary tarball just for
> > > convenient.
> > >
> > > Olaf Flebbe <[email protected]> 於 2019年7月19日 週五 上午3:40寫道:
> > >
> > > > Hi
> > > >
> > > > Maybe we have different target groups: For educational users, trying
> to
> > > > figure out how everything works together, the single computer cluster
> > is
> > > > great.
> > > >
> > > > This Tar.gz artefact were never ever been sufficient to run in
> > production.
> > > > They did not contain 64bit libs for instance. They do not provide
> start
> > > > scripts, error prone when you accidently using the wrong java, do not
> > place
> > > > logfiles in suitable places and so on.
> > > >
> > > > This tar artefacts and instructions are POC quality, that's it.
> > > >
> > > > Please be aware that Bigtop changes directory layout, directory
> > > > permissions, configuration and startup of the original code in order
> to
> > > > support large scale installations and automation.
> > > >
> > > > We have a much better alternative: Distribution integrated
> > repositories.
> > > > Youll only have to apt/yum install hadoop-hdfs-datanode and
> everything
> > is
> > > > already setup, including java, runscripts, directory layout and
> users.
> > Then
> > > > you can concentrate on configuring that beast. You do not even need
> no
> > > > special instruction, since it is distribution native -- well, almost.
> > If
> > > > you feel that bigtops runscripts / layout is not suitable , you are
> > very
> > > > welcome to contribute.
> > > >
> > > > If i look at the instructions [1] . they contain tons of settings you
> > > > should do. With our packages you can actually do exactly this,
> without
> > > > bothering to install and configure all the dependencies.
> > > >
> > > > Olaf
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Von meinem iPad gesendet
> > > >
> > > > > Am 18.07.2019 um 08:57 schrieb Guodong Xu <[email protected]>:
> > > > >
> > > > > Hi, Evans
> > > > >
> > > > > Comments in below.
> > > > >
> > > > >>      On Tue, Jul 16, 2019 at 10:46 PM Evans Ye <
> [email protected]>
> > > > wrote:
> > > > >>
> > > > >> I'm not objecting this, but I'd love to have more discussion to
> > figure
> > > > out
> > > > >> whether this is the right thing to do. What I get from your
> > proposal is
> > > > >> users want to do things which RPM/DEB don't do well while tar.gz
> is
> > > > good at
> > > > >> it. However is it able to do it in another way which is far more
> > > > beneficial
> > > > >> in more scenarios? In general, it's like when users are asking
> for a
> > > > faster
> > > > >> horses, can we come up with cars?
> > > > >>
> > > > >> How about we start with the first step which is to elaborate why
> > users
> > > > >> choose to go for tar.gz?
> > > > >
> > > > >
> > > > > Right, agree. Here is what I learned from users of Arm servers:
> > > > >
> > > > > One background for this is, currently, CDH and Hortonworks both
> have
> > no
> > > > > official release for Arm server yet. So, Bigtop is the only
> > available and
> > > > > verified distribution to them. As you know, with effort from the
> > > > community
> > > > > and Arm Inc., Linaro, Bigtop now has officially supported Arm64.
> > > > >
> > > > > To these users, before they start to use Bigtop on Arm, they are
> > already
> > > > > familiar with each component's individual installation and usage on
> > x86.
> > > > > Most of them are released in .tar.gz format. (i.e. Most apache big
> > data
> > > > > component doesn't release in deb/rpm. So, if we tell users that the
> > only
> > > > > available format in Bigtop is deb/rpm, this just hesitates them).
> > > > >
> > > > > Eg. For Hadoop, the official site for installation is here [1] and
> > here
> > > > > [2], their release format is .tar.gz.
> > > > >
> > > > > So,
> > > > > 1. using .tar.gz is the minimum effort route for them to start
> their
> > > > touch
> > > > > with Bigtop (if we can support .tar.gz).
> > > > > 2. Bigtop has all components tested. That provides very big
> > confidence to
> > > > > the users. So they like to use the binary built from Bigtop on
> Arm64
> > > > > servers.
> > > > >
> > > > > [1]
> > > > >
> > > >
> >
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html
> > > > > [2] https://www.apache.org/dyn/closer.cgi/hadoop/common/
> > > > >
> > > > > -Guodong
> > > > >
> > > > >
> > > > >>
> > > > >> Guodong Xu <[email protected]> 於 2019年7月16日 週二 下午12:24寫道:
> > > > >>
> > > > >>> Hi, all
> > > > >>>
> > > > >>> This is a request for comments for a potential new feature.
> Please
> > > > have a
> > > > >>> look and let me know whether this adds value to the community.
> > > > Appreciate
> > > > >>> your opinions on this.
> > > > >>>
> > > > >>> As most of you know already, so far, Bigtop provides two
> packaging
> > > > >> format:
> > > > >>> deb and rpm. Users are recommended to install through apt and yum
> > for
> > > > >> their
> > > > >>> respective linux distributions.
> > > > >>>
> > > > >>> Yet, there are still users coming to me and asking for the more
> > > > >> traditional
> > > > >>> .tar.gz packages. I know that shouldn't be a problem for x86
> users,
> > > > since
> > > > >>> they can always download .tar.gz bin releases from each
> component's
> > > > >>> official release website. But for users from other architectures,
> > like
> > > > >>> Arm64 and powerpc, story is different. .tar.gz are not readily
> > > > available
> > > > >> to
> > > > >>> these architectures.
> > > > >>>
> > > > >>> In Bigtop, we are doing the job of building/packaging/testing big
> > data
> > > > >>> components in one single place. So, it makes sense that we step
> > in, and
> > > > >>> during our process of building each component, we keep a binary
> > .tar.gz
> > > > >>> copy for each component we supported.
> > > > >>>
> > > > >>> If you are supportive to the above idea, then next step will be
> > how to
> > > > >>> achieve that. I did some research actually. So, my method is to
> do
> > two
> > > > >>> things:
> > > > >>> 0. this can be a subsequent task of either $target-deb or
> > $target-rpm.
> > > > It
> > > > >>> depends on one of them two.
> > > > >>> 1. add tar and cp into each component's 'do-component-build'
> file.
> > > > >>> 2. add a new set of packaging tasks for each component. This
> means
> > to
> > > > >> add a
> > > > >>> new task in genTasks() in packages.gradle.
> > > > >>>  Something like
> > > > >>> Task t = task "${target}-bin-tar" (
> > > > >>> description: "Building .tar.gz binary for $target artifacts",
> > > > >>> group: PACKAGES_GROUP) doLast { ...
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> PS:
> > > > >>> An example in 'do-component-build' for hadoop. Other components
> > > > similar.
> > > > >>>
> > > > >>> diff --git a/bigtop-packages/src/common/hadoop/do-component-build
> > > > >>> b/bigtop-packages/src/common/hadoop/do-component-build
> > > > >>> index 2a1a6345..cb8ddd9d 100644
> > > > >>> --- a/bigtop-packages/src/common/hadoop/do-component-build
> > > > >>> +++ b/bigtop-packages/src/common/hadoop/do-component-build
> > > > >>> @@ -146,3 +146,6 @@ cp -r target/staging/hadoop-project
> > build/share/doc
> > > > >>>
> > > > >>> # Copy fuse output to the build directory
> > > > >>> cp
> > > > >>>
> > > > >>>
> > > > >>
> > > >
> >
> hadoop-hdfs-project/hadoop-hdfs-native-client/target/main/native/fuse-dfs/fuse_dfs
> > > > >>> build/bin
> > > > >>> +
> > > > >>> +# Copy binary build result
> > > > >>> +cp hadoop-dist/target/hadoop-${HADOOP_VERSION}.tar.gz ..
> > > > >>>
> > > > >>>
> > > > >>> Best regards,
> > > > >>> -Guodong
> > > > >>>
> > > > >>
> > > >
> >
>


-- 
IRC: ganeshraju@#linaro on irc.freenode.ne <http://irc.freenode.net/>t

Reply via email to