Hey all,
In an ticket (INFRA-20416) Gavin asked me if we are completely off
builds.apache.org - when I went over the jobs I've saw that
https://builds.apache.org/job/Hive-linux-ARM-trunk/ is running there once a day.
Since builds.apache.org will be shut down in sometime in the future - we should
move this job to the new ci-hadoop instance or to hive-test-kube.
The key feature of the job is that it runs the test on the "armX" slaves; which
are statically configured on b.a.o.
Not sure which way to go - but we will have to move in some direction.
cheers,
Zoltan
On 3/13/20 7:22 AM, Zhenyu Zheng wrote:
Hi Chinna,
Thanks alot for the reply, I uploaded a patch and also a github PR for
https://issues.apache.org/jira/browse/HIVE-21939 .
In the patch, I bumped the protobuf used in standalone-metadata to 2.6.1
and added a new profile, this profile will identify
the hardware architecture and if it is Aarch64, it will override the
protobuf group.id and package to com.github.os72 which
includes ARM support. For X86 platform, Hive will still download the
protobuf packages from org.google repo. I think with
this method, we can keep the influence to existing x86 users to the
minimum. I hope this could be a acceptable short-term
solution.
I've manually tested on my machine and the github PR travis CI test has
already passed, so the build process is OK, so let's
wait for the full test result from builds.apache.org.
BR,
Zhenyu
On Thu, Mar 12, 2020 at 9:23 PM Chinna Rao Lalam <lalamchinnara...@gmail.com>
wrote:
Hi Zhenyu,
Until HBase dependency resolved, without effecting the existing code on X86
i suggest create a separate profile with "os72" repo.
Down the line we should have common version for both X86 and ARM.
Hope It Helps,
Chinna
On Wed, Mar 11, 2020 at 8:39 AM Zhenyu Zheng <zhengzhenyul...@gmail.com>
wrote:
Hi Chinna, David and others might interested,
Thanks for bring this up, we are currently working on improving enabling
big-data software on the ARM platform,
we have already done fixes and providing CIs to some of the well-know
projects like:
1. Hadoop:
https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/
2. Spark: https://amplab.cs.berkeley.edu/jenkins/label/spark-arm/
3. HBase:
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Nightly-ARM/
And we are now working on projects including Hive, Kudu, etc.
Regarding to the protobuf upgrades in Hive, except upgrading to 3.x and
break dependency for HBase, there can
be some possible short-term plan(or walk-arounds), doing thes can make
Hive
work on ARM without break any
dependencies, and then we can interact with Hbase project to see how can
we
both upgrade to 3.x(since this
make take some time).
Those possible solutions can be:
1. Using pre-patched protobuf 2.5.0 with ARM support
from org.openlabtesting repo, some projects(HBase did
this: https://github.com/apache/hbase/pull/959, and we will add a
profile
for this, it will detact the host arch and
will only use this pre-patched protobuf when the host arch is ARM so that
nothing will be affected for existing x86
users; Spark and Hadoop used some other package from this repo as they
have
already upgraded protobuf).
2. Using pre-patched protobuf 2.6.1-build3 from os72's repo:
https://mvnrepository.com/artifact/com.github.os72/protoc/2.6.1-build3
Hive already used ``protoc-jar-maven-plugin`` from this repo:
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/pom.xml#L484
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/pom.xml#L472
so it is a minor update.
I've tested both ways, worked OK on the ARM machine.
So my ideas will be that we make this a two-step-work, first step we use
one of the short-term plan to enable
Hive on ARM, and 2nd Step, we look for the long run to discuss with HBase
to find out a fesiable solution.
BR,
Zhenyu Zheng
On Tue, Mar 10, 2020 at 9:29 PM Chinna Rao Lalam <
lalamchinnara...@gmail.com>
wrote:
Thanks David for the quick reply.
Yes currently HBase protobuf version mismatch is the issue.
We are investigating any common version of protobuf is there to support
for
HBase an HIVE.
On Tue, Mar 10, 2020 at 6:08 PM David Lavati
<dlav...@cloudera.com.invalid
wrote:
Hi,
I recently investigated this in
https://issues.apache.org/jira/browse/HIVE-20359 and it didn't look
feasible, as protobuf has to be in sync across the whole stack, but
the
external protobuf version for hbase is on 2.5.0 for hbase-2.x, see my
comment on the jira and
https://github.com/apache/hbase/blob/branch-2.2/pom.xml#L1414.
Kind Regards,
David
On Tue, Mar 10, 2020 at 1:11 PM Chinna Rao Lalam <
lalamchinnara...@gmail.com>
wrote:
Hi All,
We are working on HIVE building and running on ARM architecture. As
part
of
this i want to add one periodic HIVE JENKINS JOB on ARM machines.
Which
helps to identity gaps for ARM, If any issues identified we will
keep
on
working to fix those issues.
Currently to run HIVE on aarch64, we need to upgrade to
protobuf-3.7.1.
HIVE-21939 <https://issues.apache.org/jira/browse/HIVE-21939>
upgraded
protobuf-2.5.0 to protobuf-3.7.1, It have mainly 2 failures. We are
working
on this.
Thanks,
Chinna Rao Lalam
--
Hope It Helps,
Chinna
--
Hope It Helps,
Chinna
