Thank you all:) The details of the arm64 test job as below: 1. The job runs on instance which flavor is 16vCPUs | 32GB, using Ubuntu 18.04 64bit with ARM, with a 500GB root volume 2. The job runs once every day on utc 12:30 3. The job script general is ./bin/bootstrap_development.sh & bin/run-all-tests.sh you can find the script https://github.com/theopenlab/openlab-zuul-jobs/blob/master/playbooks/impala-build-test-arm64/run.yaml 4. we send the job results to bui...@impala.apache.org
On Sat, Oct 10, 2020 at 5:33 AM Tim Armstrong <tarmstr...@cloudera.com> wrote: > It came through ok - > https://lists.apache.org/list.html?bui...@impala.apache.org > > On Thu, Oct 8, 2020 at 6:42 PM Tianhua huang <huangtianhua...@gmail.com> > wrote: > > > Hi Jim and Tim, > > > > Thank you very much. > > I am back from vacation, sorry to reply so late, I will configure the arm > > job to send the test result to bui...@impala.apache.org today. > > > > On Tue, Oct 6, 2020 at 7:50 AM Jim Apple <apa...@jbapple.com> wrote: > > > > > I did a test on EC2 Graviton 2, and I was able to start from a blank > > Ubuntu > > > 18.04 image and get all the way to all tests passing. This *included > > > building the native-toolchain from scratch, with no manual input > > required* > > > - it just bootstrapped with no fuss! I used a r6g.2xlarge, which has a > > list > > > price of 40 cents per hour, though you have to pay for EBS, too. If we > > get > > > to a place where we want this to be in the gerrit merge step, I expect > it > > > to be smooth sailing. > > > > > > I'm really impressed with those of you who have worked on the pieces of > > > this to put it together! Thank you for your contributions to the > > community! > > > > > > Here, roughly, are how one could duplicate my test: > > > > > > 1. Launch an r6g.2xlarge using Ubuntu 18.04, Arm64, with a 500GB > root > > > volume. Leave the following ports open: > > > - $YOUR_IP, udp 60000-61000 for mosh, in both IPV4 and IPv6 > > > - $YOUR_IP, tcp 22 for ssh, in both IPV4 and IPv6 > > > - All TCP and UDP from itself, for loopback > > > 2. ssh in and > > > - apt-get update > > > - apt-get install tmux git mosh > > > 3. Terminate your ssh session and connect again, this time with mosh > > > 4. Start a tmux session > > > 5. git clone https://github.com/apache/impala > > > 6. cd impala > > > 7. export IMPALA_HOME=$(pwd) > > > 8. ./bin/bootstrap_development.sh && ./bin/run-all-tests.sh > > > > > > I'd love to see a performance comparison between two different > equal-cost > > > clusters on AWS: one Arm64, one x86-64, but I'll leave that to the > > testers > > > with larger budgets of both time and money! :-D > > > > > > On Fri, Oct 2, 2020 at 11:05 AM Jim Apple <apa...@jbapple.com> wrote: > > > > > > > SG; submitted request for builds@ > > > > > > > > Tim, I made you and I moderators. You're welcome to decline. :-) > > > > Volunteers are welcome, but I expect zero moderation work - I set the > > > list > > > > to "subscribers can post; all others are rejected", so no moderation > > > > messages should show up. > > > > > > > > The form said it will be acted on in 24 hours and the PMC will be > > > informed > > > > when the mailing list is ready. > > > > > > > > Also, FWIW, aside from Docker, AWS supports some ARM EC2 types. > > > > > > > > On Thu, Oct 1, 2020 at 4:22 PM Tim Armstrong < > tarmstr...@cloudera.com> > > > > wrote: > > > > > > > >> I think we got stalled here. I never got anywhere with configuring > > > >> commits@ > > > >> to accept emails from openlab and not sure that it's possible. > > > >> > > > >> In order to get something going, my suggestion is to copy what Kudu > > did > > > >> and > > > >> have this go to a custom mailing list. If you follow the thread here > > it > > > >> describes what they did > > > >> > > > >> > > > > > > https://lists.apache.org/thread.html/rfcd230882ec34171462a0c8c53ef6e8efcad50c9d67f8da18a508d07%40%3Cdev.kudu.apache.org%3E > > > >> > > > >> Jim, it looks like the PMC chair is needed to create a new mailing > > list > > > >> via > > > >> https://selfserve.apache.org/ (see > > > >> https://issues.apache.org/jira/browse/INFRA-20604), so maybe we can > > go > > > >> ahead with that route? > > > >> > > > >> - Tim > > > >> > > > >> I think it would be good to run the tests on impala.jenkins.io but > I > > > >> don't > > > >> want to hold up things further. Another interesting technology is > the > > > ARM > > > >> emulation support in Docker. > > > >> > > > >> On Sun, Sep 20, 2020 at 8:53 PM Jim Apple <jbap...@apache.org> > wrote: > > > >> > > > >> > I'm very excited about this development! I don't see any jobs at > > > >> > status.openlabtesting.org, though -- is > > > >> > > > > >> > > > > http://status.openlabtesting.org/builds?job_name=impala-build-test-arm64 > > > >> > the right link? Maybe only the owner of a job can see it? > > > >> > > > > >> > As far as the price, based on https://ec2instances.info/, you > might > > > be > > > >> > able > > > >> > to eyeball the cost. I'm no longer a Jenkins admin, so I'll let > the > > > >> other > > > >> > community members who are admins speak more to this, if it seems > > > >> > feasible to them. > > > >> > > > > >> > On Sat, Sep 19, 2020 at 2:31 AM Tianhua huang < > > > >> huangtianhua...@gmail.com> > > > >> > wrote: > > > >> > > > > >> > > I am sorry there is no API with Openlab. But we use ZUUL in our > > > >> system, > > > >> > you > > > >> > > can call ZUUL api to get the job information, > > > >> > > simple like: curl > > > >> > > > > > >> > > > > > >> > > > > >> > > > > > > http://status.openlabtesting.org/api/builds?job_name=impala-build-test-arm64 > > > >> > > | python -m json.tool > > > >> > > The result we will get is jobs info of > 'impala-build-test-arm64', > > > >> > including > > > >> > > job status, job log url and so on, I am not sure if we can > > integrate > > > >> with > > > >> > > the job info into impala.jenkins.io? Then the arm job still > runs > > > >> > triggered > > > >> > > by OpenLab, and jenkins.impala.io only to get the result and > show > > > >> > > the result in it. > > > >> > > > > > >> > > Not sure it's worth doing this, maybe let > > comm...@impala.apache.org > > > >> > allow > > > >> > > receiving the result email from Openlab is more easy:) > > > >> > > > > > >> > > > > > >> > > > > > >> > > On Sat, Sep 19, 2020 at 11:03 AM Shant Hovsepian < > > > >> > > sh...@superdupershant.com> > > > >> > > wrote: > > > >> > > > > > >> > > > Would it be possible to setup an API Key or webhook with > > > >> > openlabstesting? > > > >> > > > > > > >> > > > Then we'd have an Impala Jenkins job that uses the API to call > > > >> > > > openlabstesting to do the build and report the results back. > > > >> > > > > > > >> > > > -Shant > > > >> > > > > > > >> > > > On Fri, Sep 18, 2020 at 10:24 PM Tianhua huang < > > > >> > > huangtianhua...@gmail.com> > > > >> > > > wrote: > > > >> > > > > > > >> > > > > Hi Shant, > > > >> > > > > > > > >> > > > > Thanks for your reply, maybe it's a good idea to trigger arm > > > >> testing > > > >> > by > > > >> > > > > jenkins.impala.io, we can donate an arm instance, but I > don't > > > >> know > > > >> > how > > > >> > > > to > > > >> > > > > do it. Maybe the Jenkins owner of Impala can help me? > > > >> > > > > > > > >> > > > > > > > >> > > > > On Sat, Sep 19, 2020 at 2:46 AM Shant Hovsepian < > > > >> > > > sh...@superdupershant.com > > > >> > > > > > > > > >> > > > > wrote: > > > >> > > > > > > > >> > > > > > Great work Tianhua! > > > >> > > > > > > > > >> > > > > > This is a very exciting enhancement. > > > >> > > > > > > > > >> > > > > > dev@ isn't usually meant for automated test run output. > > Would > > > >> it > > > >> > be > > > >> > > > > > possible to get the openlabtesting runs triggered by > > > >> > > jenkins.impala.io > > > >> > > > > > then > > > >> > > > > > have the results of the test run come from that jenkins > > > >> instance? > > > >> > > This > > > >> > > > > > should solve the email problem as the notification can > come > > > from > > > >> > the > > > >> > > > > > jenkins account and still give us one centralized place to > > > look > > > >> at > > > >> > > > > various > > > >> > > > > > test statuses. > > > >> > > > > > > > > >> > > > > > -Shant > > > >> > > > > > > > > >> > > > > > On Thu, Sep 17, 2020 at 8:08 AM Tianhua huang < > > > >> > > > huangtianhua...@gmail.com > > > >> > > > > > > > > >> > > > > > wrote: > > > >> > > > > > > > > >> > > > > > > Hi all, > > > >> > > > > > > > > > >> > > > > > > Impala can run/test on arm64 platform after our efforts, > > the > > > >> > issues > > > >> > > > we > > > >> > > > > > > fixed you can find in > > > >> > > > > https://issues.apache.org/jira/browse/IMPALA-9376 > > > >> > > > > > > > > > >> > > > > > > We are particularly grateful to Tim and Quanlong for > their > > > >> help, > > > >> > > > thanks > > > >> > > > > > > very much! > > > >> > > > > > > > > > >> > > > > > > And I have integrated Impala arm64 CI with OpenLab( > > > >> > > > > > > https://openlabtesting.org/ , a community to do open > > source > > > >> > > project > > > >> > > > > > > testing), > > > >> > > > > > > see > > > >> > > > > > > > > > >> > > > > > > > >> > > > > > >> > > > > http://status.openlabtesting.org/builds?job_name=impala-build-test-arm64 > > > >> > > > > > > the > > > >> > > > > > > periodic test job runs for a few days already and runs > > once > > > >> per > > > >> > > day. > > > >> > > > > > > > > > >> > > > > > > Tim suggested us to send the arm64 test result to > > > >> > > > > > > comm...@impala.apache.org, > > > >> > > > > > > but seems it can receive the emails only ncome from @ > > > >> apache.org. > > > >> > > > > > > Now I wonder if we can send the test result to Impala > > > >> community > > > >> > > > > maillist > > > >> > > > > > > here dev@impala.apache.org, to tell developers that > > Impala > > > >> can > > > >> > > runs > > > >> > > > on > > > >> > > > > > > arm64, to find the compatibility in time. > > > >> > > > > > > > > > >> > > > > > > What do you think? Or maybe there is any other good > > > >> suggestion? > > > >> > > > > > > > > > >> > > > > > > Waiting for your reply. > > > >> > > > > > > > > > >> > > > > > > Thank you very much! > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > > > > >
