Hi Chinna, David and others might interested, Thanks for bring this up, we are currently working on improving enabling big-data software on the ARM platform, we have already done fixes and providing CIs to some of the well-know projects like: 1. Hadoop: https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/ 2. Spark: https://amplab.cs.berkeley.edu/jenkins/label/spark-arm/ 3. HBase: https://builds.apache.org/view/H-L/view/HBase/job/HBase-Nightly-ARM/
And we are now working on projects including Hive, Kudu, etc. Regarding to the protobuf upgrades in Hive, except upgrading to 3.x and break dependency for HBase, there can be some possible short-term plan(or walk-arounds), doing thes can make Hive work on ARM without break any dependencies, and then we can interact with Hbase project to see how can we both upgrade to 3.x(since this make take some time). Those possible solutions can be: 1. Using pre-patched protobuf 2.5.0 with ARM support from org.openlabtesting repo, some projects(HBase did this: https://github.com/apache/hbase/pull/959, and we will add a profile for this, it will detact the host arch and will only use this pre-patched protobuf when the host arch is ARM so that nothing will be affected for existing x86 users; Spark and Hadoop used some other package from this repo as they have already upgraded protobuf). 2. Using pre-patched protobuf 2.6.1-build3 from os72's repo: https://mvnrepository.com/artifact/com.github.os72/protoc/2.6.1-build3 Hive already used ``protoc-jar-maven-plugin`` from this repo: https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/pom.xml#L484 https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/pom.xml#L472 so it is a minor update. I've tested both ways, worked OK on the ARM machine. So my ideas will be that we make this a two-step-work, first step we use one of the short-term plan to enable Hive on ARM, and 2nd Step, we look for the long run to discuss with HBase to find out a fesiable solution. BR, Zhenyu Zheng On Tue, Mar 10, 2020 at 9:29 PM Chinna Rao Lalam <lalamchinnara...@gmail.com> wrote: > Thanks David for the quick reply. > Yes currently HBase protobuf version mismatch is the issue. > > We are investigating any common version of protobuf is there to support for > HBase an HIVE. > > > > > On Tue, Mar 10, 2020 at 6:08 PM David Lavati <dlav...@cloudera.com.invalid > > > wrote: > > > Hi, > > > > I recently investigated this in > > https://issues.apache.org/jira/browse/HIVE-20359 and it didn't look > > feasible, as protobuf has to be in sync across the whole stack, but the > > external protobuf version for hbase is on 2.5.0 for hbase-2.x, see my > > comment on the jira and > > https://github.com/apache/hbase/blob/branch-2.2/pom.xml#L1414. > > > > Kind Regards, > > David > > > > On Tue, Mar 10, 2020 at 1:11 PM Chinna Rao Lalam < > > lalamchinnara...@gmail.com> > > wrote: > > > > > Hi All, > > > > > > We are working on HIVE building and running on ARM architecture. As > part > > of > > > this i want to add one periodic HIVE JENKINS JOB on ARM machines. Which > > > helps to identity gaps for ARM, If any issues identified we will keep > on > > > working to fix those issues. > > > > > > Currently to run HIVE on aarch64, we need to upgrade to protobuf-3.7.1. > > > HIVE-21939 <https://issues.apache.org/jira/browse/HIVE-21939> upgraded > > > protobuf-2.5.0 to protobuf-3.7.1, It have mainly 2 failures. We are > > working > > > on this. > > > > > > Thanks, > > > Chinna Rao Lalam > > > > > > > > -- > Hope It Helps, > Chinna >
