Hi Ashutosh, I want to migrate https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/ this job to the new system https://ci-hadoop.apache.org. Please grant me rights to create jobs on the new Jenkins https://ci-builds.apache.org. My LDAP userid : chinnaraol. I have requested the same in apache infra. Thank you.
Thank you Stamatis,Zoltan for the pointers. Thanks, Chinna On Thu, Jun 18, 2020 at 5:36 PM Stamatis Zampetakis <zabe...@gmail.com> wrote: > Hello Chinna, > > The hudson-jobadmin privilege can be granted by PMC chairs. > I don't know if there is any particular policy in Hive on who should have > this privilege so I guess you should request it from Ashutosh. > > Best, > Stamatis > > On Thu, Jun 18, 2020 at 12:05 PM Zoltan Haindrich <k...@rxd.hu> wrote: > >> Hey Chinna! >> >> On 6/18/20 11:43 AM, Chinna Rao Lalam wrote: >> > As you said, migrating this job to the new ci-hadoop instance looks >> good as >> > Hadoop also shares the same armN slaves. >> >> Sounds great! >> >> > I am able to login the new ci-hadoop instance with Apache LDAP >> credentials, >> > but i am not able to see the job creation option. Should I request >> access >> > or the process for creation of a job is different than jenkin?. >> > Please guide me to create the new job in the ci-hadoop instance. I will >> > migrate this job after connecting the armN slaves to the new system. >> >> >> I've also logged in - and apparently I've create job rights; I'm happy to >> help, but the best would be to self-service yourselft :) >> I think you may miss the "hudson-jobadmin" privilege. >> Probably Gavin (or someone on the infra team) could help you with that.. >> to talk to them quickly - you can reach them on the #asfinfra channel (on >> the asf-slack). >> >> The migration effort is coordinated thru the hadoop-migrations mailing >> list (I've cc-ed that list) >> you may want to subscribe to it by sending a mail to: >> hadoop-migrations-subscr...@infra.apache.org >> >> cheers, >> Zoltan >> >> >> >> > >> > Thanks >> > Chinna >> > >> > On Wed, Jun 17, 2020 at 11:57 AM Zhenyu Zheng < >> zhengzhenyul...@gmail.com> >> > wrote: >> > >> >> Hi Zoltan, >> >> >> >> Thanks alot for the information, so looks like one possible solution >> is as >> >> you suggest, move the current ARM2 and ARM3 (those two were donate to >> >> builds.apache.org by us) to the new ci-hadoop cluster and set up the >> jobs >> >> just as what has been done in current jenkins. >> >> >> >> I will also ask our team member works on other projects to find out >> what >> >> the status of other projects is. >> >> >> >> BR, >> >> >> >> On Tue, Jun 16, 2020 at 6:41 PM Zoltan Haindrich <k...@rxd.hu> wrote: >> >> >> >>> Hey, >> >>> >> >>> There is an effort by the Apache Infra to change the way Jenkins >> stuff is >> >>> organized; a couple months ago Gavin wrote an email about it: >> >>> >> >>> >> http://mail-archives.apache.org/mod_mbox/tez-dev/202004.mbox/%3ccan0gg1dodepzatjz9bofe-2ver7qg7h0hmvyjmsldgjr8_r...@mail.gmail.com%3E >> >>> The resources for running these jobs are coming from the H0~H21 slaves >> >>> which will be migrated to the new jenkins master eventually. >> >>> >> >>> >> So please >> >>> >> suggest a way which direction we can move and can you share some >> >>> details >> >>> >> about the new ci-hadoop instance. >> >>> >> >>> Since Hadoop testing is also happening on ARM - I think the best >> would be >> >>> to also migrate the armN slaves and the Hive arm nightly over to the >> new >> >>> ci-hadoop instance. >> >>> >> >>> On 6/16/20 8:40 AM, Zhenyu Zheng wrote: >> >>>> Thanks for the info, I wonder if where does the resource of ci-hadoop >> >>> and >> >>>> hive-test-kube come from? Do they include ARM resources? >> >>> >> >>> Interesting question; the resources for Hive testing are donated by >> >>> Cloudera. >> >>> About the ARM workers I think Chinna could provide more details. >> >>> ...I've no idea don't know who sponsors the Hxx slaves >> >>> >> >>>> Can you provide some more information about how the new >> hive-test-kube >> >>> is >> >>>> running? >> >>> It's basically a Jenkins instance which is using kubernetes pods to >> run >> >>> things. >> >>> The whole thing is running on a GKE cluster. >> >>> While I was working on it I collected stuff needed for it in this >> repo: >> >>> https://github.com/kgyrtkirk/hive-test-kube/ >> >>> it should be possible to start a new deployment using that stuff >> >>> >> >>> cheers, >> >>> Zoltan >> >>> >> >>>> >> >>>> BR, >> >>>> Kevin Zheng >> >>>> >> >>>> On Tue, Jun 16, 2020 at 12:41 PM Chinna Rao Lalam < >> >>>> lalamchinnara...@gmail.com> wrote: >> >>>> >> >>>>> Hi Zoltan, >> >>>>> >> >>>>> Thanks for the update. >> >>>>> >> >>>>> Current https://builds.apache.org/job/Hive-linux-ARM-trunk/ job is >> >>>>> targeting to run hive tests daily on "arm" slaves, it is using 2 arm >> >>>>> slaves. >> >>>>> To find any potential issues with "arm" and fix the issues. So >> please >> >>>>> suggest a way which direction we can move and can you share some >> >>> details >> >>>>> about the new ci-hadoop instance. >> >>>>> >> >>>>> Thanks, >> >>>>> Chinna >> >>>>> >> >>>>> On Mon, Jun 15, 2020 at 3:56 PM Zoltan Haindrich <k...@rxd.hu> >> wrote: >> >>>>> >> >>>>>> Hey all, >> >>>>>> >> >>>>>> In an ticket (INFRA-20416) Gavin asked me if we are completely off >> >>>>>> builds.apache.org - when I went over the jobs I've saw that >> >>>>>> https://builds.apache.org/job/Hive-linux-ARM-trunk/ is running >> there >> >>>>>> once a day. >> >>>>>> >> >>>>>> Since builds.apache.org will be shut down in sometime in the >> future >> >>> - we >> >>>>>> should move this job to the new ci-hadoop instance or to >> >>> hive-test-kube. >> >>>>>> The key feature of the job is that it runs the test on the "armX" >> >>> slaves; >> >>>>>> which are statically configured on b.a.o. >> >>>>>> Not sure which way to go - but we will have to move in some >> direction. >> >>>>>> >> >>>>>> cheers, >> >>>>>> Zoltan >> >>>>>> >> >>>>>> >> >>>>>> On 3/13/20 7:22 AM, Zhenyu Zheng wrote: >> >>>>>>> Hi Chinna, >> >>>>>>> >> >>>>>>> Thanks alot for the reply, I uploaded a patch and also a github PR >> >>> for >> >>>>>>> https://issues.apache.org/jira/browse/HIVE-21939 . >> >>>>>>> In the patch, I bumped the protobuf used in standalone-metadata to >> >>> 2.6.1 >> >>>>>>> and added a new profile, this profile will identify >> >>>>>>> the hardware architecture and if it is Aarch64, it will override >> the >> >>>>>>> protobuf group.id and package to com.github.os72 which >> >>>>>>> includes ARM support. For X86 platform, Hive will still download >> the >> >>>>>>> protobuf packages from org.google repo. I think with >> >>>>>>> this method, we can keep the influence to existing x86 users to >> the >> >>>>>>> minimum. I hope this could be a acceptable short-term >> >>>>>>> solution. >> >>>>>>> >> >>>>>>> I've manually tested on my machine and the github PR travis CI >> test >> >>> has >> >>>>>>> already passed, so the build process is OK, so let's >> >>>>>>> wait for the full test result from builds.apache.org. >> >>>>>>> >> >>>>>>> BR, >> >>>>>>> >> >>>>>>> Zhenyu >> >>>>>>> >> >>>>>>> On Thu, Mar 12, 2020 at 9:23 PM Chinna Rao Lalam < >> >>>>>> lalamchinnara...@gmail.com> >> >>>>>>> wrote: >> >>>>>>> >> >>>>>>>> Hi Zhenyu, >> >>>>>>>> >> >>>>>>>> Until HBase dependency resolved, without effecting the existing >> code >> >>>>>> on X86 >> >>>>>>>> i suggest create a separate profile with "os72" repo. >> >>>>>>>> >> >>>>>>>> Down the line we should have common version for both X86 and ARM. >> >>>>>>>> >> >>>>>>>> Hope It Helps, >> >>>>>>>> Chinna >> >>>>>>>> >> >>>>>>>> On Wed, Mar 11, 2020 at 8:39 AM Zhenyu Zheng < >> >>>>>> zhengzhenyul...@gmail.com> >> >>>>>>>> wrote: >> >>>>>>>> >> >>>>>>>>> Hi Chinna, David and others might interested, >> >>>>>>>>> >> >>>>>>>>> Thanks for bring this up, we are currently working on improving >> >>>>>> enabling >> >>>>>>>>> big-data software on the ARM platform, >> >>>>>>>>> we have already done fixes and providing CIs to some of the >> >>> well-know >> >>>>>>>>> projects like: >> >>>>>>>>> 1. Hadoop: >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>>>> >> >>> >> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/ >> >>>>>>>>> 2. Spark: >> https://amplab.cs.berkeley.edu/jenkins/label/spark-arm/ >> >>>>>>>>> 3. HBase: >> >>>>>>>>> >> >>> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Nightly-ARM/ >> >>>>>>>>> >> >>>>>>>>> And we are now working on projects including Hive, Kudu, etc. >> >>>>>>>>> >> >>>>>>>>> Regarding to the protobuf upgrades in Hive, except upgrading to >> 3.x >> >>>>>> and >> >>>>>>>>> break dependency for HBase, there can >> >>>>>>>>> be some possible short-term plan(or walk-arounds), doing thes >> can >> >>> make >> >>>>>>>> Hive >> >>>>>>>>> work on ARM without break any >> >>>>>>>>> dependencies, and then we can interact with Hbase project to see >> >>> how >> >>>>>> can >> >>>>>>>> we >> >>>>>>>>> both upgrade to 3.x(since this >> >>>>>>>>> make take some time). >> >>>>>>>>> >> >>>>>>>>> Those possible solutions can be: >> >>>>>>>>> 1. Using pre-patched protobuf 2.5.0 with ARM support >> >>>>>>>>> from org.openlabtesting repo, some projects(HBase did >> >>>>>>>>> this: https://github.com/apache/hbase/pull/959, and we will >> add a >> >>>>>>>> profile >> >>>>>>>>> for this, it will detact the host arch and >> >>>>>>>>> will only use this pre-patched protobuf when the host arch is >> ARM >> >>> so >> >>>>>> that >> >>>>>>>>> nothing will be affected for existing x86 >> >>>>>>>>> users; Spark and Hadoop used some other package from this repo >> as >> >>> they >> >>>>>>>> have >> >>>>>>>>> already upgraded protobuf). >> >>>>>>>>> >> >>>>>>>>> 2. Using pre-patched protobuf 2.6.1-build3 from os72's repo: >> >>>>>>>>> >> >>>>>> >> >>> >> https://mvnrepository.com/artifact/com.github.os72/protoc/2.6.1-build3 >> >>>>>>>>> Hive already used ``protoc-jar-maven-plugin`` from this repo: >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>>>> >> >>> >> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/pom.xml#L484 >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>>>> >> >>> >> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/pom.xml#L472 >> >>>>>>>>> so it is a minor update. >> >>>>>>>>> >> >>>>>>>>> I've tested both ways, worked OK on the ARM machine. >> >>>>>>>>> >> >>>>>>>>> So my ideas will be that we make this a two-step-work, first >> step >> >>> we >> >>>>>> use >> >>>>>>>>> one of the short-term plan to enable >> >>>>>>>>> Hive on ARM, and 2nd Step, we look for the long run to discuss >> with >> >>>>>> HBase >> >>>>>>>>> to find out a fesiable solution. >> >>>>>>>>> >> >>>>>>>>> BR, >> >>>>>>>>> >> >>>>>>>>> Zhenyu Zheng >> >>>>>>>>> >> >>>>>>>>> On Tue, Mar 10, 2020 at 9:29 PM Chinna Rao Lalam < >> >>>>>>>>> lalamchinnara...@gmail.com> >> >>>>>>>>> wrote: >> >>>>>>>>> >> >>>>>>>>>> Thanks David for the quick reply. >> >>>>>>>>>> Yes currently HBase protobuf version mismatch is the issue. >> >>>>>>>>>> >> >>>>>>>>>> We are investigating any common version of protobuf is there to >> >>>>>> support >> >>>>>>>>> for >> >>>>>>>>>> HBase an HIVE. >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> On Tue, Mar 10, 2020 at 6:08 PM David Lavati >> >>>>>>>>> <dlav...@cloudera.com.invalid >> >>>>>>>>>>> >> >>>>>>>>>> wrote: >> >>>>>>>>>> >> >>>>>>>>>>> Hi, >> >>>>>>>>>>> >> >>>>>>>>>>> I recently investigated this in >> >>>>>>>>>>> https://issues.apache.org/jira/browse/HIVE-20359 and it >> didn't >> >>> look >> >>>>>>>>>>> feasible, as protobuf has to be in sync across the whole >> stack, >> >>> but >> >>>>>>>> the >> >>>>>>>>>>> external protobuf version for hbase is on 2.5.0 for hbase-2.x, >> >>> see >> >>>>>> my >> >>>>>>>>>>> comment on the jira and >> >>>>>>>>>>> https://github.com/apache/hbase/blob/branch-2.2/pom.xml#L1414 >> . >> >>>>>>>>>>> >> >>>>>>>>>>> Kind Regards, >> >>>>>>>>>>> David >> >>>>>>>>>>> >> >>>>>>>>>>> On Tue, Mar 10, 2020 at 1:11 PM Chinna Rao Lalam < >> >>>>>>>>>>> lalamchinnara...@gmail.com> >> >>>>>>>>>>> wrote: >> >>>>>>>>>>> >> >>>>>>>>>>>> Hi All, >> >>>>>>>>>>>> >> >>>>>>>>>>>> We are working on HIVE building and running on ARM >> >>> architecture. As >> >>>>>>>>>> part >> >>>>>>>>>>> of >> >>>>>>>>>>>> this i want to add one periodic HIVE JENKINS JOB on ARM >> >>> machines. >> >>>>>>>>> Which >> >>>>>>>>>>>> helps to identity gaps for ARM, If any issues identified we >> will >> >>>>>>>> keep >> >>>>>>>>>> on >> >>>>>>>>>>>> working to fix those issues. >> >>>>>>>>>>>> >> >>>>>>>>>>>> Currently to run HIVE on aarch64, we need to upgrade to >> >>>>>>>>> protobuf-3.7.1. >> >>>>>>>>>>>> HIVE-21939 <https://issues.apache.org/jira/browse/HIVE-21939 >> > >> >>>>>>>>> upgraded >> >>>>>>>>>>>> protobuf-2.5.0 to protobuf-3.7.1, It have mainly 2 failures. >> We >> >>> are >> >>>>>>>>>>> working >> >>>>>>>>>>>> on this. >> >>>>>>>>>>>> >> >>>>>>>>>>>> Thanks, >> >>>>>>>>>>>> Chinna Rao Lalam >> >>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> -- >> >>>>>>>>>> Hope It Helps, >> >>>>>>>>>> Chinna >> >>>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> -- >> >>>>>>>> Hope It Helps, >> >>>>>>>> Chinna >> >>>>>>>> >> >>>>>>> >> >>>>>> >> >>>>> >> >>>>> >> >>>>> -- >> >>>>> Hope It Helps, >> >>>>> Chinna >> >>>>> >> >>>> >> >>> >> >> >> > >> > -- Hope It Helps, Chinna
