Hi Chinna, Thanks alot for the reply, I uploaded a patch and also a github PR for https://issues.apache.org/jira/browse/HIVE-21939 . In the patch, I bumped the protobuf used in standalone-metadata to 2.6.1 and added a new profile, this profile will identify the hardware architecture and if it is Aarch64, it will override the protobuf group.id and package to com.github.os72 which includes ARM support. For X86 platform, Hive will still download the protobuf packages from org.google repo. I think with this method, we can keep the influence to existing x86 users to the minimum. I hope this could be a acceptable short-term solution.
I've manually tested on my machine and the github PR travis CI test has already passed, so the build process is OK, so let's wait for the full test result from builds.apache.org. BR, Zhenyu On Thu, Mar 12, 2020 at 9:23 PM Chinna Rao Lalam <lalamchinnara...@gmail.com> wrote: > Hi Zhenyu, > > Until HBase dependency resolved, without effecting the existing code on X86 > i suggest create a separate profile with "os72" repo. > > Down the line we should have common version for both X86 and ARM. > > Hope It Helps, > Chinna > > On Wed, Mar 11, 2020 at 8:39 AM Zhenyu Zheng <zhengzhenyul...@gmail.com> > wrote: > > > Hi Chinna, David and others might interested, > > > > Thanks for bring this up, we are currently working on improving enabling > > big-data software on the ARM platform, > > we have already done fixes and providing CIs to some of the well-know > > projects like: > > 1. Hadoop: > > > > > https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/ > > 2. Spark: https://amplab.cs.berkeley.edu/jenkins/label/spark-arm/ > > 3. HBase: > > https://builds.apache.org/view/H-L/view/HBase/job/HBase-Nightly-ARM/ > > > > And we are now working on projects including Hive, Kudu, etc. > > > > Regarding to the protobuf upgrades in Hive, except upgrading to 3.x and > > break dependency for HBase, there can > > be some possible short-term plan(or walk-arounds), doing thes can make > Hive > > work on ARM without break any > > dependencies, and then we can interact with Hbase project to see how can > we > > both upgrade to 3.x(since this > > make take some time). > > > > Those possible solutions can be: > > 1. Using pre-patched protobuf 2.5.0 with ARM support > > from org.openlabtesting repo, some projects(HBase did > > this: https://github.com/apache/hbase/pull/959, and we will add a > profile > > for this, it will detact the host arch and > > will only use this pre-patched protobuf when the host arch is ARM so that > > nothing will be affected for existing x86 > > users; Spark and Hadoop used some other package from this repo as they > have > > already upgraded protobuf). > > > > 2. Using pre-patched protobuf 2.6.1-build3 from os72's repo: > > https://mvnrepository.com/artifact/com.github.os72/protoc/2.6.1-build3 > > Hive already used ``protoc-jar-maven-plugin`` from this repo: > > > > > https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/pom.xml#L484 > > > > > > > https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/pom.xml#L472 > > so it is a minor update. > > > > I've tested both ways, worked OK on the ARM machine. > > > > So my ideas will be that we make this a two-step-work, first step we use > > one of the short-term plan to enable > > Hive on ARM, and 2nd Step, we look for the long run to discuss with HBase > > to find out a fesiable solution. > > > > BR, > > > > Zhenyu Zheng > > > > On Tue, Mar 10, 2020 at 9:29 PM Chinna Rao Lalam < > > lalamchinnara...@gmail.com> > > wrote: > > > > > Thanks David for the quick reply. > > > Yes currently HBase protobuf version mismatch is the issue. > > > > > > We are investigating any common version of protobuf is there to support > > for > > > HBase an HIVE. > > > > > > > > > > > > > > > On Tue, Mar 10, 2020 at 6:08 PM David Lavati > > <dlav...@cloudera.com.invalid > > > > > > > wrote: > > > > > > > Hi, > > > > > > > > I recently investigated this in > > > > https://issues.apache.org/jira/browse/HIVE-20359 and it didn't look > > > > feasible, as protobuf has to be in sync across the whole stack, but > the > > > > external protobuf version for hbase is on 2.5.0 for hbase-2.x, see my > > > > comment on the jira and > > > > https://github.com/apache/hbase/blob/branch-2.2/pom.xml#L1414. > > > > > > > > Kind Regards, > > > > David > > > > > > > > On Tue, Mar 10, 2020 at 1:11 PM Chinna Rao Lalam < > > > > lalamchinnara...@gmail.com> > > > > wrote: > > > > > > > > > Hi All, > > > > > > > > > > We are working on HIVE building and running on ARM architecture. As > > > part > > > > of > > > > > this i want to add one periodic HIVE JENKINS JOB on ARM machines. > > Which > > > > > helps to identity gaps for ARM, If any issues identified we will > keep > > > on > > > > > working to fix those issues. > > > > > > > > > > Currently to run HIVE on aarch64, we need to upgrade to > > protobuf-3.7.1. > > > > > HIVE-21939 <https://issues.apache.org/jira/browse/HIVE-21939> > > upgraded > > > > > protobuf-2.5.0 to protobuf-3.7.1, It have mainly 2 failures. We are > > > > working > > > > > on this. > > > > > > > > > > Thanks, > > > > > Chinna Rao Lalam > > > > > > > > > > > > > > > > > > -- > > > Hope It Helps, > > > Chinna > > > > > > > > -- > Hope It Helps, > Chinna >
