Hi Zoltan, Chinna, Thanks for the info, I wonder if where does the resource of ci-hadoop and hive-test-kube come from? Do they include ARM resources? Can you provide some more information about how the new hive-test-kube is running?
BR, Kevin Zheng On Tue, Jun 16, 2020 at 12:41 PM Chinna Rao Lalam < lalamchinnara...@gmail.com> wrote: > Hi Zoltan, > > Thanks for the update. > > Current https://builds.apache.org/job/Hive-linux-ARM-trunk/ job is > targeting to run hive tests daily on "arm" slaves, it is using 2 arm > slaves. > To find any potential issues with "arm" and fix the issues. So please > suggest a way which direction we can move and can you share some details > about the new ci-hadoop instance. > > Thanks, > Chinna > > On Mon, Jun 15, 2020 at 3:56 PM Zoltan Haindrich <k...@rxd.hu> wrote: > >> Hey all, >> >> In an ticket (INFRA-20416) Gavin asked me if we are completely off >> builds.apache.org - when I went over the jobs I've saw that >> https://builds.apache.org/job/Hive-linux-ARM-trunk/ is running there >> once a day. >> >> Since builds.apache.org will be shut down in sometime in the future - we >> should move this job to the new ci-hadoop instance or to hive-test-kube. >> The key feature of the job is that it runs the test on the "armX" slaves; >> which are statically configured on b.a.o. >> Not sure which way to go - but we will have to move in some direction. >> >> cheers, >> Zoltan >> >> >> On 3/13/20 7:22 AM, Zhenyu Zheng wrote: >> > Hi Chinna, >> > >> > Thanks alot for the reply, I uploaded a patch and also a github PR for >> > https://issues.apache.org/jira/browse/HIVE-21939 . >> > In the patch, I bumped the protobuf used in standalone-metadata to 2.6.1 >> > and added a new profile, this profile will identify >> > the hardware architecture and if it is Aarch64, it will override the >> > protobuf group.id and package to com.github.os72 which >> > includes ARM support. For X86 platform, Hive will still download the >> > protobuf packages from org.google repo. I think with >> > this method, we can keep the influence to existing x86 users to the >> > minimum. I hope this could be a acceptable short-term >> > solution. >> > >> > I've manually tested on my machine and the github PR travis CI test has >> > already passed, so the build process is OK, so let's >> > wait for the full test result from builds.apache.org. >> > >> > BR, >> > >> > Zhenyu >> > >> > On Thu, Mar 12, 2020 at 9:23 PM Chinna Rao Lalam < >> lalamchinnara...@gmail.com> >> > wrote: >> > >> >> Hi Zhenyu, >> >> >> >> Until HBase dependency resolved, without effecting the existing code >> on X86 >> >> i suggest create a separate profile with "os72" repo. >> >> >> >> Down the line we should have common version for both X86 and ARM. >> >> >> >> Hope It Helps, >> >> Chinna >> >> >> >> On Wed, Mar 11, 2020 at 8:39 AM Zhenyu Zheng < >> zhengzhenyul...@gmail.com> >> >> wrote: >> >> >> >>> Hi Chinna, David and others might interested, >> >>> >> >>> Thanks for bring this up, we are currently working on improving >> enabling >> >>> big-data software on the ARM platform, >> >>> we have already done fixes and providing CIs to some of the well-know >> >>> projects like: >> >>> 1. Hadoop: >> >>> >> >>> >> >> >> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/ >> >>> 2. Spark: https://amplab.cs.berkeley.edu/jenkins/label/spark-arm/ >> >>> 3. HBase: >> >>> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Nightly-ARM/ >> >>> >> >>> And we are now working on projects including Hive, Kudu, etc. >> >>> >> >>> Regarding to the protobuf upgrades in Hive, except upgrading to 3.x >> and >> >>> break dependency for HBase, there can >> >>> be some possible short-term plan(or walk-arounds), doing thes can make >> >> Hive >> >>> work on ARM without break any >> >>> dependencies, and then we can interact with Hbase project to see how >> can >> >> we >> >>> both upgrade to 3.x(since this >> >>> make take some time). >> >>> >> >>> Those possible solutions can be: >> >>> 1. Using pre-patched protobuf 2.5.0 with ARM support >> >>> from org.openlabtesting repo, some projects(HBase did >> >>> this: https://github.com/apache/hbase/pull/959, and we will add a >> >> profile >> >>> for this, it will detact the host arch and >> >>> will only use this pre-patched protobuf when the host arch is ARM so >> that >> >>> nothing will be affected for existing x86 >> >>> users; Spark and Hadoop used some other package from this repo as they >> >> have >> >>> already upgraded protobuf). >> >>> >> >>> 2. Using pre-patched protobuf 2.6.1-build3 from os72's repo: >> >>> >> https://mvnrepository.com/artifact/com.github.os72/protoc/2.6.1-build3 >> >>> Hive already used ``protoc-jar-maven-plugin`` from this repo: >> >>> >> >>> >> >> >> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/pom.xml#L484 >> >>> >> >>> >> >>> >> >> >> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/pom.xml#L472 >> >>> so it is a minor update. >> >>> >> >>> I've tested both ways, worked OK on the ARM machine. >> >>> >> >>> So my ideas will be that we make this a two-step-work, first step we >> use >> >>> one of the short-term plan to enable >> >>> Hive on ARM, and 2nd Step, we look for the long run to discuss with >> HBase >> >>> to find out a fesiable solution. >> >>> >> >>> BR, >> >>> >> >>> Zhenyu Zheng >> >>> >> >>> On Tue, Mar 10, 2020 at 9:29 PM Chinna Rao Lalam < >> >>> lalamchinnara...@gmail.com> >> >>> wrote: >> >>> >> >>>> Thanks David for the quick reply. >> >>>> Yes currently HBase protobuf version mismatch is the issue. >> >>>> >> >>>> We are investigating any common version of protobuf is there to >> support >> >>> for >> >>>> HBase an HIVE. >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> On Tue, Mar 10, 2020 at 6:08 PM David Lavati >> >>> <dlav...@cloudera.com.invalid >> >>>>> >> >>>> wrote: >> >>>> >> >>>>> Hi, >> >>>>> >> >>>>> I recently investigated this in >> >>>>> https://issues.apache.org/jira/browse/HIVE-20359 and it didn't look >> >>>>> feasible, as protobuf has to be in sync across the whole stack, but >> >> the >> >>>>> external protobuf version for hbase is on 2.5.0 for hbase-2.x, see >> my >> >>>>> comment on the jira and >> >>>>> https://github.com/apache/hbase/blob/branch-2.2/pom.xml#L1414. >> >>>>> >> >>>>> Kind Regards, >> >>>>> David >> >>>>> >> >>>>> On Tue, Mar 10, 2020 at 1:11 PM Chinna Rao Lalam < >> >>>>> lalamchinnara...@gmail.com> >> >>>>> wrote: >> >>>>> >> >>>>>> Hi All, >> >>>>>> >> >>>>>> We are working on HIVE building and running on ARM architecture. As >> >>>> part >> >>>>> of >> >>>>>> this i want to add one periodic HIVE JENKINS JOB on ARM machines. >> >>> Which >> >>>>>> helps to identity gaps for ARM, If any issues identified we will >> >> keep >> >>>> on >> >>>>>> working to fix those issues. >> >>>>>> >> >>>>>> Currently to run HIVE on aarch64, we need to upgrade to >> >>> protobuf-3.7.1. >> >>>>>> HIVE-21939 <https://issues.apache.org/jira/browse/HIVE-21939> >> >>> upgraded >> >>>>>> protobuf-2.5.0 to protobuf-3.7.1, It have mainly 2 failures. We are >> >>>>> working >> >>>>>> on this. >> >>>>>> >> >>>>>> Thanks, >> >>>>>> Chinna Rao Lalam >> >>>>>> >> >>>>> >> >>>> >> >>>> >> >>>> -- >> >>>> Hope It Helps, >> >>>> Chinna >> >>>> >> >>> >> >> >> >> >> >> -- >> >> Hope It Helps, >> >> Chinna >> >> >> > >> > > > -- > Hope It Helps, > Chinna >
