Hi Zoltan, Thanks for the update.
Current https://builds.apache.org/job/Hive-linux-ARM-trunk/ job is targeting to run hive tests daily on "arm" slaves, it is using 2 arm slaves. To find any potential issues with "arm" and fix the issues. So please suggest a way which direction we can move and can you share some details about the new ci-hadoop instance. Thanks, Chinna On Mon, Jun 15, 2020 at 3:56 PM Zoltan Haindrich <k...@rxd.hu> wrote: > Hey all, > > In an ticket (INFRA-20416) Gavin asked me if we are completely off > builds.apache.org - when I went over the jobs I've saw that > https://builds.apache.org/job/Hive-linux-ARM-trunk/ is running there once > a day. > > Since builds.apache.org will be shut down in sometime in the future - we > should move this job to the new ci-hadoop instance or to hive-test-kube. > The key feature of the job is that it runs the test on the "armX" slaves; > which are statically configured on b.a.o. > Not sure which way to go - but we will have to move in some direction. > > cheers, > Zoltan > > > On 3/13/20 7:22 AM, Zhenyu Zheng wrote: > > Hi Chinna, > > > > Thanks alot for the reply, I uploaded a patch and also a github PR for > > https://issues.apache.org/jira/browse/HIVE-21939 . > > In the patch, I bumped the protobuf used in standalone-metadata to 2.6.1 > > and added a new profile, this profile will identify > > the hardware architecture and if it is Aarch64, it will override the > > protobuf group.id and package to com.github.os72 which > > includes ARM support. For X86 platform, Hive will still download the > > protobuf packages from org.google repo. I think with > > this method, we can keep the influence to existing x86 users to the > > minimum. I hope this could be a acceptable short-term > > solution. > > > > I've manually tested on my machine and the github PR travis CI test has > > already passed, so the build process is OK, so let's > > wait for the full test result from builds.apache.org. > > > > BR, > > > > Zhenyu > > > > On Thu, Mar 12, 2020 at 9:23 PM Chinna Rao Lalam < > lalamchinnara...@gmail.com> > > wrote: > > > >> Hi Zhenyu, > >> > >> Until HBase dependency resolved, without effecting the existing code on > X86 > >> i suggest create a separate profile with "os72" repo. > >> > >> Down the line we should have common version for both X86 and ARM. > >> > >> Hope It Helps, > >> Chinna > >> > >> On Wed, Mar 11, 2020 at 8:39 AM Zhenyu Zheng <zhengzhenyul...@gmail.com > > > >> wrote: > >> > >>> Hi Chinna, David and others might interested, > >>> > >>> Thanks for bring this up, we are currently working on improving > enabling > >>> big-data software on the ARM platform, > >>> we have already done fixes and providing CIs to some of the well-know > >>> projects like: > >>> 1. Hadoop: > >>> > >>> > >> > https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/ > >>> 2. Spark: https://amplab.cs.berkeley.edu/jenkins/label/spark-arm/ > >>> 3. HBase: > >>> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Nightly-ARM/ > >>> > >>> And we are now working on projects including Hive, Kudu, etc. > >>> > >>> Regarding to the protobuf upgrades in Hive, except upgrading to 3.x and > >>> break dependency for HBase, there can > >>> be some possible short-term plan(or walk-arounds), doing thes can make > >> Hive > >>> work on ARM without break any > >>> dependencies, and then we can interact with Hbase project to see how > can > >> we > >>> both upgrade to 3.x(since this > >>> make take some time). > >>> > >>> Those possible solutions can be: > >>> 1. Using pre-patched protobuf 2.5.0 with ARM support > >>> from org.openlabtesting repo, some projects(HBase did > >>> this: https://github.com/apache/hbase/pull/959, and we will add a > >> profile > >>> for this, it will detact the host arch and > >>> will only use this pre-patched protobuf when the host arch is ARM so > that > >>> nothing will be affected for existing x86 > >>> users; Spark and Hadoop used some other package from this repo as they > >> have > >>> already upgraded protobuf). > >>> > >>> 2. Using pre-patched protobuf 2.6.1-build3 from os72's repo: > >>> https://mvnrepository.com/artifact/com.github.os72/protoc/2.6.1-build3 > >>> Hive already used ``protoc-jar-maven-plugin`` from this repo: > >>> > >>> > >> > https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/pom.xml#L484 > >>> > >>> > >>> > >> > https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/pom.xml#L472 > >>> so it is a minor update. > >>> > >>> I've tested both ways, worked OK on the ARM machine. > >>> > >>> So my ideas will be that we make this a two-step-work, first step we > use > >>> one of the short-term plan to enable > >>> Hive on ARM, and 2nd Step, we look for the long run to discuss with > HBase > >>> to find out a fesiable solution. > >>> > >>> BR, > >>> > >>> Zhenyu Zheng > >>> > >>> On Tue, Mar 10, 2020 at 9:29 PM Chinna Rao Lalam < > >>> lalamchinnara...@gmail.com> > >>> wrote: > >>> > >>>> Thanks David for the quick reply. > >>>> Yes currently HBase protobuf version mismatch is the issue. > >>>> > >>>> We are investigating any common version of protobuf is there to > support > >>> for > >>>> HBase an HIVE. > >>>> > >>>> > >>>> > >>>> > >>>> On Tue, Mar 10, 2020 at 6:08 PM David Lavati > >>> <dlav...@cloudera.com.invalid > >>>>> > >>>> wrote: > >>>> > >>>>> Hi, > >>>>> > >>>>> I recently investigated this in > >>>>> https://issues.apache.org/jira/browse/HIVE-20359 and it didn't look > >>>>> feasible, as protobuf has to be in sync across the whole stack, but > >> the > >>>>> external protobuf version for hbase is on 2.5.0 for hbase-2.x, see my > >>>>> comment on the jira and > >>>>> https://github.com/apache/hbase/blob/branch-2.2/pom.xml#L1414. > >>>>> > >>>>> Kind Regards, > >>>>> David > >>>>> > >>>>> On Tue, Mar 10, 2020 at 1:11 PM Chinna Rao Lalam < > >>>>> lalamchinnara...@gmail.com> > >>>>> wrote: > >>>>> > >>>>>> Hi All, > >>>>>> > >>>>>> We are working on HIVE building and running on ARM architecture. As > >>>> part > >>>>> of > >>>>>> this i want to add one periodic HIVE JENKINS JOB on ARM machines. > >>> Which > >>>>>> helps to identity gaps for ARM, If any issues identified we will > >> keep > >>>> on > >>>>>> working to fix those issues. > >>>>>> > >>>>>> Currently to run HIVE on aarch64, we need to upgrade to > >>> protobuf-3.7.1. > >>>>>> HIVE-21939 <https://issues.apache.org/jira/browse/HIVE-21939> > >>> upgraded > >>>>>> protobuf-2.5.0 to protobuf-3.7.1, It have mainly 2 failures. We are > >>>>> working > >>>>>> on this. > >>>>>> > >>>>>> Thanks, > >>>>>> Chinna Rao Lalam > >>>>>> > >>>>> > >>>> > >>>> > >>>> -- > >>>> Hope It Helps, > >>>> Chinna > >>>> > >>> > >> > >> > >> -- > >> Hope It Helps, > >> Chinna > >> > > > -- Hope It Helps, Chinna
