guyuqi commented on code in PR #1053: URL: https://github.com/apache/bigtop/pull/1053#discussion_r1031008728
########## bigtop-packages/src/common/phoenix/patch1-phoenix-hbase-compat.diff: ########## @@ -0,0 +1,69 @@ +diff --git a/phoenix-hbase-compat-2.4.1/pom.xml b/phoenix-hbase-compat-2.4.1/pom.xml +index de397d869..ead8bcf9c 100644 +--- a/phoenix-hbase-compat-2.4.1/pom.xml ++++ b/phoenix-hbase-compat-2.4.1/pom.xml +@@ -31,7 +31,7 @@ + <description>Compatibility module for HBase 2.4.1+</description> + + <properties> +- <hbase24.compat.version>2.4.1</hbase24.compat.version> ++ <hbase24.compat.version>2.4.13</hbase24.compat.version> + </properties> + + <dependencies> +diff --git a/pom.xml b/pom.xml +index b0a36925e..7a7887170 100644 +--- a/pom.xml ++++ b/pom.xml +@@ -80,10 +80,11 @@ + <!-- Hadoop and Hbase-thirdparty version --> + <!-- These are expected to be overridden to conform to cluster versions + along with hbase.version (defined in profiles) --> +- <hadoop.version>3.1.4</hadoop.version> +- <hbase.thirdparty.version>2.2.1</hbase.thirdparty.version> ++ <hadoop.version>3.3.4</hadoop.version> Review Comment: Hi @iwasakims > Should the `hadoop.version` be 3.2.4 rather than 3.3.4 to aligh with [mpacks expecting Bigtop 3.1 stack](https://issues.apache.org/jira/browse/BIGTOP-3689)? The `Hadoop verison` would be 3.3.4 for the new coming Bigtop-3.2 release. If we use 3.1 release packages (hadoop-3.2.4), it would look weird that new Bigtop release is 3.2 but the Mpack is 3.1. The users could just only deploy Bigtop-3.1 release packages, not the new Bigtop-3.2 packages. Acctually, all work in BIGTOP-3689 is to support Bigtop-3.2 Mpack. > How about using Apache release of Hadoop and HBase as done in https://github.com/apache/ambari-metrics/pull/69 ? If ambari-metrics is using its own embedded HBase, it is not necessary to be Bigtop one. I've tested the related patches (https://github.com/apache/ambari-metrics/pull/69) and it failed to deploy Ambari metrics on the Mpack cluster. It seems they hadn't been fully tested in Bigtop + Ambari + Ambari metrics deployment.  ### The Root Cause: In the comming Bigtop-3.2 (`hadoop-3.3.4`/`HBase-2.13`/`Phoenix-5.12` ...): The versions of the embeded components in `Ambari`/ `Ambari metrrics` should also be same with the versions in Mpack services/Bigtop components respectively. Otherwise, if we deploy `hadoop-3.3.4` and the embeded Hadoop in Ambari metrics is 3.1.1, there are some conflicts between Hadoop and Ambari metrics from their `hadoop.metrics2.sink.timeline` Class interface. The embeded `phoenix-5.0.0-HBase-2.0` and `hbase-2.0.2-bin` are also out of date which were released in 2020. It makes sense to update them to the latest ones (Bigtop-3.2). Consequently, we should provide the latest Bigtop packages for ambari metrics. In Bigtop-3.2 release, we will offer the latest rpms/debs of Bigdata components. And these latest components could be easily deployed by Ambari Mpack. I built latest Bigtop packages and setup a temporary file server in my local environment for Bigtop-3.2 + Ambari Mpack development.  The related [repo info](https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/bigtop-ambari-mpack/bgtp-ambari-mpack/src/main/resources/stacks/BGTP/1.0/repos/repoinfo.xml#L21) would be updated before 3.2 release. ``` <repo> <baseurl>http://repos.bigtop.apache.org/releases/3.2.0/centos/7/$basearch</baseurl> </repo> ``` Combined with the PRs: `Add Ambari Metrics to Mpack` - https://github.com/apache/bigtop/pull/1056, `Adjust Ambari-2.7.5 dependency` - https://github.com/apache/bigtop/pull/1054, our latest Bigtop services could be successfully deployed by Bigtop + Ambari-2.7.5 + Ambari metrics (data visualization).   **Let's go back to the beginning, we still need to find a place (Bigtop public repo) to store these Binary-TARs as Hortonworks did for Ambari.** ``` hadoop-3.3.4.tar.gz hbase-2.4.13-bin.tar.gz phoenix-core-5.1.2.jar phoenix-hbase-2.4-5.1.2-bin.tar.gz phoenix-hbase-compat-2.4.1-5.1.2.jar ``` ### HOWTO Deploy: 1. `cd bigtop/provisioner/bigtop-stack-provisioner/docker/centos7 ` 2. Build environment image:`./build-image.sh` 3. Setup Environment:`./build-containers.sh` 4. Access Ambari webui: `localhost:8080` 5. Deploy Bigtop serivces -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
