Hi Jim and Tim, Thank you very much. I am back from vacation, sorry to reply so late, I will configure the arm job to send the test result to bui...@impala.apache.org today.
On Tue, Oct 6, 2020 at 7:50 AM Jim Apple <apa...@jbapple.com> wrote: > I did a test on EC2 Graviton 2, and I was able to start from a blank Ubuntu > 18.04 image and get all the way to all tests passing. This *included > building the native-toolchain from scratch, with no manual input required* > - it just bootstrapped with no fuss! I used a r6g.2xlarge, which has a list > price of 40 cents per hour, though you have to pay for EBS, too. If we get > to a place where we want this to be in the gerrit merge step, I expect it > to be smooth sailing. > > I'm really impressed with those of you who have worked on the pieces of > this to put it together! Thank you for your contributions to the community! > > Here, roughly, are how one could duplicate my test: > > 1. Launch an r6g.2xlarge using Ubuntu 18.04, Arm64, with a 500GB root > volume. Leave the following ports open: > - $YOUR_IP, udp 60000-61000 for mosh, in both IPV4 and IPv6 > - $YOUR_IP, tcp 22 for ssh, in both IPV4 and IPv6 > - All TCP and UDP from itself, for loopback > 2. ssh in and > - apt-get update > - apt-get install tmux git mosh > 3. Terminate your ssh session and connect again, this time with mosh > 4. Start a tmux session > 5. git clone https://github.com/apache/impala > 6. cd impala > 7. export IMPALA_HOME=$(pwd) > 8. ./bin/bootstrap_development.sh && ./bin/run-all-tests.sh > > I'd love to see a performance comparison between two different equal-cost > clusters on AWS: one Arm64, one x86-64, but I'll leave that to the testers > with larger budgets of both time and money! :-D > > On Fri, Oct 2, 2020 at 11:05 AM Jim Apple <apa...@jbapple.com> wrote: > > > SG; submitted request for builds@ > > > > Tim, I made you and I moderators. You're welcome to decline. :-) > > Volunteers are welcome, but I expect zero moderation work - I set the > list > > to "subscribers can post; all others are rejected", so no moderation > > messages should show up. > > > > The form said it will be acted on in 24 hours and the PMC will be > informed > > when the mailing list is ready. > > > > Also, FWIW, aside from Docker, AWS supports some ARM EC2 types. > > > > On Thu, Oct 1, 2020 at 4:22 PM Tim Armstrong <tarmstr...@cloudera.com> > > wrote: > > > >> I think we got stalled here. I never got anywhere with configuring > >> commits@ > >> to accept emails from openlab and not sure that it's possible. > >> > >> In order to get something going, my suggestion is to copy what Kudu did > >> and > >> have this go to a custom mailing list. If you follow the thread here it > >> describes what they did > >> > >> > https://lists.apache.org/thread.html/rfcd230882ec34171462a0c8c53ef6e8efcad50c9d67f8da18a508d07%40%3Cdev.kudu.apache.org%3E > >> > >> Jim, it looks like the PMC chair is needed to create a new mailing list > >> via > >> https://selfserve.apache.org/ (see > >> https://issues.apache.org/jira/browse/INFRA-20604), so maybe we can go > >> ahead with that route? > >> > >> - Tim > >> > >> I think it would be good to run the tests on impala.jenkins.io but I > >> don't > >> want to hold up things further. Another interesting technology is the > ARM > >> emulation support in Docker. > >> > >> On Sun, Sep 20, 2020 at 8:53 PM Jim Apple <jbap...@apache.org> wrote: > >> > >> > I'm very excited about this development! I don't see any jobs at > >> > status.openlabtesting.org, though -- is > >> > > >> > http://status.openlabtesting.org/builds?job_name=impala-build-test-arm64 > >> > the right link? Maybe only the owner of a job can see it? > >> > > >> > As far as the price, based on https://ec2instances.info/, you might > be > >> > able > >> > to eyeball the cost. I'm no longer a Jenkins admin, so I'll let the > >> other > >> > community members who are admins speak more to this, if it seems > >> > feasible to them. > >> > > >> > On Sat, Sep 19, 2020 at 2:31 AM Tianhua huang < > >> huangtianhua...@gmail.com> > >> > wrote: > >> > > >> > > I am sorry there is no API with Openlab. But we use ZUUL in our > >> system, > >> > you > >> > > can call ZUUL api to get the job information, > >> > > simple like: curl > >> > > > >> > > > >> > > >> > http://status.openlabtesting.org/api/builds?job_name=impala-build-test-arm64 > >> > > | python -m json.tool > >> > > The result we will get is jobs info of 'impala-build-test-arm64', > >> > including > >> > > job status, job log url and so on, I am not sure if we can integrate > >> with > >> > > the job info into impala.jenkins.io? Then the arm job still runs > >> > triggered > >> > > by OpenLab, and jenkins.impala.io only to get the result and show > >> > > the result in it. > >> > > > >> > > Not sure it's worth doing this, maybe let comm...@impala.apache.org > >> > allow > >> > > receiving the result email from Openlab is more easy:) > >> > > > >> > > > >> > > > >> > > On Sat, Sep 19, 2020 at 11:03 AM Shant Hovsepian < > >> > > sh...@superdupershant.com> > >> > > wrote: > >> > > > >> > > > Would it be possible to setup an API Key or webhook with > >> > openlabstesting? > >> > > > > >> > > > Then we'd have an Impala Jenkins job that uses the API to call > >> > > > openlabstesting to do the build and report the results back. > >> > > > > >> > > > -Shant > >> > > > > >> > > > On Fri, Sep 18, 2020 at 10:24 PM Tianhua huang < > >> > > huangtianhua...@gmail.com> > >> > > > wrote: > >> > > > > >> > > > > Hi Shant, > >> > > > > > >> > > > > Thanks for your reply, maybe it's a good idea to trigger arm > >> testing > >> > by > >> > > > > jenkins.impala.io, we can donate an arm instance, but I don't > >> know > >> > how > >> > > > to > >> > > > > do it. Maybe the Jenkins owner of Impala can help me? > >> > > > > > >> > > > > > >> > > > > On Sat, Sep 19, 2020 at 2:46 AM Shant Hovsepian < > >> > > > sh...@superdupershant.com > >> > > > > > > >> > > > > wrote: > >> > > > > > >> > > > > > Great work Tianhua! > >> > > > > > > >> > > > > > This is a very exciting enhancement. > >> > > > > > > >> > > > > > dev@ isn't usually meant for automated test run output. Would > >> it > >> > be > >> > > > > > possible to get the openlabtesting runs triggered by > >> > > jenkins.impala.io > >> > > > > > then > >> > > > > > have the results of the test run come from that jenkins > >> instance? > >> > > This > >> > > > > > should solve the email problem as the notification can come > from > >> > the > >> > > > > > jenkins account and still give us one centralized place to > look > >> at > >> > > > > various > >> > > > > > test statuses. > >> > > > > > > >> > > > > > -Shant > >> > > > > > > >> > > > > > On Thu, Sep 17, 2020 at 8:08 AM Tianhua huang < > >> > > > huangtianhua...@gmail.com > >> > > > > > > >> > > > > > wrote: > >> > > > > > > >> > > > > > > Hi all, > >> > > > > > > > >> > > > > > > Impala can run/test on arm64 platform after our efforts, the > >> > issues > >> > > > we > >> > > > > > > fixed you can find in > >> > > > > https://issues.apache.org/jira/browse/IMPALA-9376 > >> > > > > > > > >> > > > > > > We are particularly grateful to Tim and Quanlong for their > >> help, > >> > > > thanks > >> > > > > > > very much! > >> > > > > > > > >> > > > > > > And I have integrated Impala arm64 CI with OpenLab( > >> > > > > > > https://openlabtesting.org/ , a community to do open source > >> > > project > >> > > > > > > testing), > >> > > > > > > see > >> > > > > > > > >> > > > > > >> > > > >> > http://status.openlabtesting.org/builds?job_name=impala-build-test-arm64 > >> > > > > > > the > >> > > > > > > periodic test job runs for a few days already and runs once > >> per > >> > > day. > >> > > > > > > > >> > > > > > > Tim suggested us to send the arm64 test result to > >> > > > > > > comm...@impala.apache.org, > >> > > > > > > but seems it can receive the emails only ncome from @ > >> apache.org. > >> > > > > > > Now I wonder if we can send the test result to Impala > >> community > >> > > > > maillist > >> > > > > > > here dev@impala.apache.org, to tell developers that Impala > >> can > >> > > runs > >> > > > on > >> > > > > > > arm64, to find the compatibility in time. > >> > > > > > > > >> > > > > > > What do you think? Or maybe there is any other good > >> suggestion? > >> > > > > > > > >> > > > > > > Waiting for your reply. > >> > > > > > > > >> > > > > > > Thank you very much! > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > >
