>From technical perspective Olaf's points are reasonable. However I'd like to bridge the gap here so that we can strike a balance of what arm folks need.
Guodong let's back the story with some technical discussions. For example, is it because your customer already implemented their deployment tool with tar.gz? If so that's a strong reason from their perspective. To me tar.gz has no much pros for production. For Apache projects, all the release are source code hence tar.gz is the most simple way. Furthermore, most of the projects will provide compiled binary tarball just for convenient. Olaf Flebbe <[email protected]> 於 2019年7月19日 週五 上午3:40寫道: > Hi > > Maybe we have different target groups: For educational users, trying to > figure out how everything works together, the single computer cluster is > great. > > This Tar.gz artefact were never ever been sufficient to run in production. > They did not contain 64bit libs for instance. They do not provide start > scripts, error prone when you accidently using the wrong java, do not place > logfiles in suitable places and so on. > > This tar artefacts and instructions are POC quality, that's it. > > Please be aware that Bigtop changes directory layout, directory > permissions, configuration and startup of the original code in order to > support large scale installations and automation. > > We have a much better alternative: Distribution integrated repositories. > Youll only have to apt/yum install hadoop-hdfs-datanode and everything is > already setup, including java, runscripts, directory layout and users. Then > you can concentrate on configuring that beast. You do not even need no > special instruction, since it is distribution native -- well, almost. If > you feel that bigtops runscripts / layout is not suitable , you are very > welcome to contribute. > > If i look at the instructions [1] . they contain tons of settings you > should do. With our packages you can actually do exactly this, without > bothering to install and configure all the dependencies. > > Olaf > > > > > > Von meinem iPad gesendet > > > Am 18.07.2019 um 08:57 schrieb Guodong Xu <[email protected]>: > > > > Hi, Evans > > > > Comments in below. > > > >> On Tue, Jul 16, 2019 at 10:46 PM Evans Ye <[email protected]> > wrote: > >> > >> I'm not objecting this, but I'd love to have more discussion to figure > out > >> whether this is the right thing to do. What I get from your proposal is > >> users want to do things which RPM/DEB don't do well while tar.gz is > good at > >> it. However is it able to do it in another way which is far more > beneficial > >> in more scenarios? In general, it's like when users are asking for a > faster > >> horses, can we come up with cars? > >> > >> How about we start with the first step which is to elaborate why users > >> choose to go for tar.gz? > > > > > > Right, agree. Here is what I learned from users of Arm servers: > > > > One background for this is, currently, CDH and Hortonworks both have no > > official release for Arm server yet. So, Bigtop is the only available and > > verified distribution to them. As you know, with effort from the > community > > and Arm Inc., Linaro, Bigtop now has officially supported Arm64. > > > > To these users, before they start to use Bigtop on Arm, they are already > > familiar with each component's individual installation and usage on x86. > > Most of them are released in .tar.gz format. (i.e. Most apache big data > > component doesn't release in deb/rpm. So, if we tell users that the only > > available format in Bigtop is deb/rpm, this just hesitates them). > > > > Eg. For Hadoop, the official site for installation is here [1] and here > > [2], their release format is .tar.gz. > > > > So, > > 1. using .tar.gz is the minimum effort route for them to start their > touch > > with Bigtop (if we can support .tar.gz). > > 2. Bigtop has all components tested. That provides very big confidence to > > the users. So they like to use the binary built from Bigtop on Arm64 > > servers. > > > > [1] > > > https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html > > [2] https://www.apache.org/dyn/closer.cgi/hadoop/common/ > > > > -Guodong > > > > > >> > >> Guodong Xu <[email protected]> 於 2019年7月16日 週二 下午12:24寫道: > >> > >>> Hi, all > >>> > >>> This is a request for comments for a potential new feature. Please > have a > >>> look and let me know whether this adds value to the community. > Appreciate > >>> your opinions on this. > >>> > >>> As most of you know already, so far, Bigtop provides two packaging > >> format: > >>> deb and rpm. Users are recommended to install through apt and yum for > >> their > >>> respective linux distributions. > >>> > >>> Yet, there are still users coming to me and asking for the more > >> traditional > >>> .tar.gz packages. I know that shouldn't be a problem for x86 users, > since > >>> they can always download .tar.gz bin releases from each component's > >>> official release website. But for users from other architectures, like > >>> Arm64 and powerpc, story is different. .tar.gz are not readily > available > >> to > >>> these architectures. > >>> > >>> In Bigtop, we are doing the job of building/packaging/testing big data > >>> components in one single place. So, it makes sense that we step in, and > >>> during our process of building each component, we keep a binary .tar.gz > >>> copy for each component we supported. > >>> > >>> If you are supportive to the above idea, then next step will be how to > >>> achieve that. I did some research actually. So, my method is to do two > >>> things: > >>> 0. this can be a subsequent task of either $target-deb or $target-rpm. > It > >>> depends on one of them two. > >>> 1. add tar and cp into each component's 'do-component-build' file. > >>> 2. add a new set of packaging tasks for each component. This means to > >> add a > >>> new task in genTasks() in packages.gradle. > >>> Something like > >>> Task t = task "${target}-bin-tar" ( > >>> description: "Building .tar.gz binary for $target artifacts", > >>> group: PACKAGES_GROUP) doLast { ... > >>> > >>> > >>> > >>> PS: > >>> An example in 'do-component-build' for hadoop. Other components > similar. > >>> > >>> diff --git a/bigtop-packages/src/common/hadoop/do-component-build > >>> b/bigtop-packages/src/common/hadoop/do-component-build > >>> index 2a1a6345..cb8ddd9d 100644 > >>> --- a/bigtop-packages/src/common/hadoop/do-component-build > >>> +++ b/bigtop-packages/src/common/hadoop/do-component-build > >>> @@ -146,3 +146,6 @@ cp -r target/staging/hadoop-project build/share/doc > >>> > >>> # Copy fuse output to the build directory > >>> cp > >>> > >>> > >> > hadoop-hdfs-project/hadoop-hdfs-native-client/target/main/native/fuse-dfs/fuse_dfs > >>> build/bin > >>> + > >>> +# Copy binary build result > >>> +cp hadoop-dist/target/hadoop-${HADOOP_VERSION}.tar.gz .. > >>> > >>> > >>> Best regards, > >>> -Guodong > >>> > >> >
