Hello Chinna, The hudson-jobadmin privilege can be granted by PMC chairs. I don't know if there is any particular policy in Hive on who should have this privilege so I guess you should request it from Ashutosh.
Best, Stamatis On Thu, Jun 18, 2020 at 12:05 PM Zoltan Haindrich <k...@rxd.hu> wrote: > Hey Chinna! > > On 6/18/20 11:43 AM, Chinna Rao Lalam wrote: > > As you said, migrating this job to the new ci-hadoop instance looks good > as > > Hadoop also shares the same armN slaves. > > Sounds great! > > > I am able to login the new ci-hadoop instance with Apache LDAP > credentials, > > but i am not able to see the job creation option. Should I request access > > or the process for creation of a job is different than jenkin?. > > Please guide me to create the new job in the ci-hadoop instance. I will > > migrate this job after connecting the armN slaves to the new system. > > > I've also logged in - and apparently I've create job rights; I'm happy to > help, but the best would be to self-service yourselft :) > I think you may miss the "hudson-jobadmin" privilege. > Probably Gavin (or someone on the infra team) could help you with that.. > to talk to them quickly - you can reach them on the #asfinfra channel (on > the asf-slack). > > The migration effort is coordinated thru the hadoop-migrations mailing > list (I've cc-ed that list) > you may want to subscribe to it by sending a mail to: > hadoop-migrations-subscr...@infra.apache.org > > cheers, > Zoltan > > > > > > > Thanks > > Chinna > > > > On Wed, Jun 17, 2020 at 11:57 AM Zhenyu Zheng <zhengzhenyul...@gmail.com > > > > wrote: > > > >> Hi Zoltan, > >> > >> Thanks alot for the information, so looks like one possible solution is > as > >> you suggest, move the current ARM2 and ARM3 (those two were donate to > >> builds.apache.org by us) to the new ci-hadoop cluster and set up the > jobs > >> just as what has been done in current jenkins. > >> > >> I will also ask our team member works on other projects to find out what > >> the status of other projects is. > >> > >> BR, > >> > >> On Tue, Jun 16, 2020 at 6:41 PM Zoltan Haindrich <k...@rxd.hu> wrote: > >> > >>> Hey, > >>> > >>> There is an effort by the Apache Infra to change the way Jenkins stuff > is > >>> organized; a couple months ago Gavin wrote an email about it: > >>> > >>> > http://mail-archives.apache.org/mod_mbox/tez-dev/202004.mbox/%3ccan0gg1dodepzatjz9bofe-2ver7qg7h0hmvyjmsldgjr8_r...@mail.gmail.com%3E > >>> The resources for running these jobs are coming from the H0~H21 slaves > >>> which will be migrated to the new jenkins master eventually. > >>> > >>> >> So please > >>> >> suggest a way which direction we can move and can you share some > >>> details > >>> >> about the new ci-hadoop instance. > >>> > >>> Since Hadoop testing is also happening on ARM - I think the best would > be > >>> to also migrate the armN slaves and the Hive arm nightly over to the > new > >>> ci-hadoop instance. > >>> > >>> On 6/16/20 8:40 AM, Zhenyu Zheng wrote: > >>>> Thanks for the info, I wonder if where does the resource of ci-hadoop > >>> and > >>>> hive-test-kube come from? Do they include ARM resources? > >>> > >>> Interesting question; the resources for Hive testing are donated by > >>> Cloudera. > >>> About the ARM workers I think Chinna could provide more details. > >>> ...I've no idea don't know who sponsors the Hxx slaves > >>> > >>>> Can you provide some more information about how the new hive-test-kube > >>> is > >>>> running? > >>> It's basically a Jenkins instance which is using kubernetes pods to run > >>> things. > >>> The whole thing is running on a GKE cluster. > >>> While I was working on it I collected stuff needed for it in this repo: > >>> https://github.com/kgyrtkirk/hive-test-kube/ > >>> it should be possible to start a new deployment using that stuff > >>> > >>> cheers, > >>> Zoltan > >>> > >>>> > >>>> BR, > >>>> Kevin Zheng > >>>> > >>>> On Tue, Jun 16, 2020 at 12:41 PM Chinna Rao Lalam < > >>>> lalamchinnara...@gmail.com> wrote: > >>>> > >>>>> Hi Zoltan, > >>>>> > >>>>> Thanks for the update. > >>>>> > >>>>> Current https://builds.apache.org/job/Hive-linux-ARM-trunk/ job is > >>>>> targeting to run hive tests daily on "arm" slaves, it is using 2 arm > >>>>> slaves. > >>>>> To find any potential issues with "arm" and fix the issues. So please > >>>>> suggest a way which direction we can move and can you share some > >>> details > >>>>> about the new ci-hadoop instance. > >>>>> > >>>>> Thanks, > >>>>> Chinna > >>>>> > >>>>> On Mon, Jun 15, 2020 at 3:56 PM Zoltan Haindrich <k...@rxd.hu> > wrote: > >>>>> > >>>>>> Hey all, > >>>>>> > >>>>>> In an ticket (INFRA-20416) Gavin asked me if we are completely off > >>>>>> builds.apache.org - when I went over the jobs I've saw that > >>>>>> https://builds.apache.org/job/Hive-linux-ARM-trunk/ is running > there > >>>>>> once a day. > >>>>>> > >>>>>> Since builds.apache.org will be shut down in sometime in the future > >>> - we > >>>>>> should move this job to the new ci-hadoop instance or to > >>> hive-test-kube. > >>>>>> The key feature of the job is that it runs the test on the "armX" > >>> slaves; > >>>>>> which are statically configured on b.a.o. > >>>>>> Not sure which way to go - but we will have to move in some > direction. > >>>>>> > >>>>>> cheers, > >>>>>> Zoltan > >>>>>> > >>>>>> > >>>>>> On 3/13/20 7:22 AM, Zhenyu Zheng wrote: > >>>>>>> Hi Chinna, > >>>>>>> > >>>>>>> Thanks alot for the reply, I uploaded a patch and also a github PR > >>> for > >>>>>>> https://issues.apache.org/jira/browse/HIVE-21939 . > >>>>>>> In the patch, I bumped the protobuf used in standalone-metadata to > >>> 2.6.1 > >>>>>>> and added a new profile, this profile will identify > >>>>>>> the hardware architecture and if it is Aarch64, it will override > the > >>>>>>> protobuf group.id and package to com.github.os72 which > >>>>>>> includes ARM support. For X86 platform, Hive will still download > the > >>>>>>> protobuf packages from org.google repo. I think with > >>>>>>> this method, we can keep the influence to existing x86 users to the > >>>>>>> minimum. I hope this could be a acceptable short-term > >>>>>>> solution. > >>>>>>> > >>>>>>> I've manually tested on my machine and the github PR travis CI test > >>> has > >>>>>>> already passed, so the build process is OK, so let's > >>>>>>> wait for the full test result from builds.apache.org. > >>>>>>> > >>>>>>> BR, > >>>>>>> > >>>>>>> Zhenyu > >>>>>>> > >>>>>>> On Thu, Mar 12, 2020 at 9:23 PM Chinna Rao Lalam < > >>>>>> lalamchinnara...@gmail.com> > >>>>>>> wrote: > >>>>>>> > >>>>>>>> Hi Zhenyu, > >>>>>>>> > >>>>>>>> Until HBase dependency resolved, without effecting the existing > code > >>>>>> on X86 > >>>>>>>> i suggest create a separate profile with "os72" repo. > >>>>>>>> > >>>>>>>> Down the line we should have common version for both X86 and ARM. > >>>>>>>> > >>>>>>>> Hope It Helps, > >>>>>>>> Chinna > >>>>>>>> > >>>>>>>> On Wed, Mar 11, 2020 at 8:39 AM Zhenyu Zheng < > >>>>>> zhengzhenyul...@gmail.com> > >>>>>>>> wrote: > >>>>>>>> > >>>>>>>>> Hi Chinna, David and others might interested, > >>>>>>>>> > >>>>>>>>> Thanks for bring this up, we are currently working on improving > >>>>>> enabling > >>>>>>>>> big-data software on the ARM platform, > >>>>>>>>> we have already done fixes and providing CIs to some of the > >>> well-know > >>>>>>>>> projects like: > >>>>>>>>> 1. Hadoop: > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>> > >>> > https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/ > >>>>>>>>> 2. Spark: > https://amplab.cs.berkeley.edu/jenkins/label/spark-arm/ > >>>>>>>>> 3. HBase: > >>>>>>>>> > >>> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Nightly-ARM/ > >>>>>>>>> > >>>>>>>>> And we are now working on projects including Hive, Kudu, etc. > >>>>>>>>> > >>>>>>>>> Regarding to the protobuf upgrades in Hive, except upgrading to > 3.x > >>>>>> and > >>>>>>>>> break dependency for HBase, there can > >>>>>>>>> be some possible short-term plan(or walk-arounds), doing thes can > >>> make > >>>>>>>> Hive > >>>>>>>>> work on ARM without break any > >>>>>>>>> dependencies, and then we can interact with Hbase project to see > >>> how > >>>>>> can > >>>>>>>> we > >>>>>>>>> both upgrade to 3.x(since this > >>>>>>>>> make take some time). > >>>>>>>>> > >>>>>>>>> Those possible solutions can be: > >>>>>>>>> 1. Using pre-patched protobuf 2.5.0 with ARM support > >>>>>>>>> from org.openlabtesting repo, some projects(HBase did > >>>>>>>>> this: https://github.com/apache/hbase/pull/959, and we will add > a > >>>>>>>> profile > >>>>>>>>> for this, it will detact the host arch and > >>>>>>>>> will only use this pre-patched protobuf when the host arch is ARM > >>> so > >>>>>> that > >>>>>>>>> nothing will be affected for existing x86 > >>>>>>>>> users; Spark and Hadoop used some other package from this repo as > >>> they > >>>>>>>> have > >>>>>>>>> already upgraded protobuf). > >>>>>>>>> > >>>>>>>>> 2. Using pre-patched protobuf 2.6.1-build3 from os72's repo: > >>>>>>>>> > >>>>>> > >>> https://mvnrepository.com/artifact/com.github.os72/protoc/2.6.1-build3 > >>>>>>>>> Hive already used ``protoc-jar-maven-plugin`` from this repo: > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>> > >>> > https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/pom.xml#L484 > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>> > >>> > https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/pom.xml#L472 > >>>>>>>>> so it is a minor update. > >>>>>>>>> > >>>>>>>>> I've tested both ways, worked OK on the ARM machine. > >>>>>>>>> > >>>>>>>>> So my ideas will be that we make this a two-step-work, first step > >>> we > >>>>>> use > >>>>>>>>> one of the short-term plan to enable > >>>>>>>>> Hive on ARM, and 2nd Step, we look for the long run to discuss > with > >>>>>> HBase > >>>>>>>>> to find out a fesiable solution. > >>>>>>>>> > >>>>>>>>> BR, > >>>>>>>>> > >>>>>>>>> Zhenyu Zheng > >>>>>>>>> > >>>>>>>>> On Tue, Mar 10, 2020 at 9:29 PM Chinna Rao Lalam < > >>>>>>>>> lalamchinnara...@gmail.com> > >>>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> Thanks David for the quick reply. > >>>>>>>>>> Yes currently HBase protobuf version mismatch is the issue. > >>>>>>>>>> > >>>>>>>>>> We are investigating any common version of protobuf is there to > >>>>>> support > >>>>>>>>> for > >>>>>>>>>> HBase an HIVE. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On Tue, Mar 10, 2020 at 6:08 PM David Lavati > >>>>>>>>> <dlav...@cloudera.com.invalid > >>>>>>>>>>> > >>>>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>>> Hi, > >>>>>>>>>>> > >>>>>>>>>>> I recently investigated this in > >>>>>>>>>>> https://issues.apache.org/jira/browse/HIVE-20359 and it didn't > >>> look > >>>>>>>>>>> feasible, as protobuf has to be in sync across the whole stack, > >>> but > >>>>>>>> the > >>>>>>>>>>> external protobuf version for hbase is on 2.5.0 for hbase-2.x, > >>> see > >>>>>> my > >>>>>>>>>>> comment on the jira and > >>>>>>>>>>> https://github.com/apache/hbase/blob/branch-2.2/pom.xml#L1414. > >>>>>>>>>>> > >>>>>>>>>>> Kind Regards, > >>>>>>>>>>> David > >>>>>>>>>>> > >>>>>>>>>>> On Tue, Mar 10, 2020 at 1:11 PM Chinna Rao Lalam < > >>>>>>>>>>> lalamchinnara...@gmail.com> > >>>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> Hi All, > >>>>>>>>>>>> > >>>>>>>>>>>> We are working on HIVE building and running on ARM > >>> architecture. As > >>>>>>>>>> part > >>>>>>>>>>> of > >>>>>>>>>>>> this i want to add one periodic HIVE JENKINS JOB on ARM > >>> machines. > >>>>>>>>> Which > >>>>>>>>>>>> helps to identity gaps for ARM, If any issues identified we > will > >>>>>>>> keep > >>>>>>>>>> on > >>>>>>>>>>>> working to fix those issues. > >>>>>>>>>>>> > >>>>>>>>>>>> Currently to run HIVE on aarch64, we need to upgrade to > >>>>>>>>> protobuf-3.7.1. > >>>>>>>>>>>> HIVE-21939 <https://issues.apache.org/jira/browse/HIVE-21939> > >>>>>>>>> upgraded > >>>>>>>>>>>> protobuf-2.5.0 to protobuf-3.7.1, It have mainly 2 failures. > We > >>> are > >>>>>>>>>>> working > >>>>>>>>>>>> on this. > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks, > >>>>>>>>>>>> Chinna Rao Lalam > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> -- > >>>>>>>>>> Hope It Helps, > >>>>>>>>>> Chinna > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> -- > >>>>>>>> Hope It Helps, > >>>>>>>> Chinna > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> Hope It Helps, > >>>>> Chinna > >>>>> > >>>> > >>> > >> > > >
