I did a test on EC2 Graviton 2, and I was able to start from a blank Ubuntu
18.04 image and get all the way to all tests passing. This *included
building the native-toolchain from scratch, with no manual input required*
- it just bootstrapped with no fuss! I used a r6g.2xlarge, which has a list
price of 40 cents per hour, though you have to pay for EBS, too. If we get
to a place where we want this to be in the gerrit merge step, I expect it
to be smooth sailing.
I'm really impressed with those of you who have worked on the pieces of
this to put it together! Thank you for your contributions to the community!
Here, roughly, are how one could duplicate my test:
1. Launch an r6g.2xlarge using Ubuntu 18.04, Arm64, with a 500GB root
volume. Leave the following ports open:
- $YOUR_IP, udp 60000-61000 for mosh, in both IPV4 and IPv6
- $YOUR_IP, tcp 22 for ssh, in both IPV4 and IPv6
- All TCP and UDP from itself, for loopback
2. ssh in and
- apt-get update
- apt-get install tmux git mosh
3. Terminate your ssh session and connect again, this time with mosh
4. Start a tmux session
5. git clone https://github.com/apache/impala
6. cd impala
7. export IMPALA_HOME=$(pwd)
8. ./bin/bootstrap_development.sh && ./bin/run-all-tests.sh
I'd love to see a performance comparison between two different equal-cost
clusters on AWS: one Arm64, one x86-64, but I'll leave that to the testers
with larger budgets of both time and money! :-D
On Fri, Oct 2, 2020 at 11:05 AM Jim Apple <apa...@jbapple.com> wrote:
> SG; submitted request for builds@
>
> Tim, I made you and I moderators. You're welcome to decline. :-)
> Volunteers are welcome, but I expect zero moderation work - I set the list
> to "subscribers can post; all others are rejected", so no moderation
> messages should show up.
>
> The form said it will be acted on in 24 hours and the PMC will be informed
> when the mailing list is ready.
>
> Also, FWIW, aside from Docker, AWS supports some ARM EC2 types.
>
> On Thu, Oct 1, 2020 at 4:22 PM Tim Armstrong <tarmstr...@cloudera.com>
> wrote:
>
>> I think we got stalled here. I never got anywhere with configuring
>> commits@
>> to accept emails from openlab and not sure that it's possible.
>>
>> In order to get something going, my suggestion is to copy what Kudu did
>> and
>> have this go to a custom mailing list. If you follow the thread here it
>> describes what they did
>>
>> https://lists.apache.org/thread.html/rfcd230882ec34171462a0c8c53ef6e8efcad50c9d67f8da18a508d07%40%3Cdev.kudu.apache.org%3E
>>
>> Jim, it looks like the PMC chair is needed to create a new mailing list
>> via
>> https://selfserve.apache.org/ (see
>> https://issues.apache.org/jira/browse/INFRA-20604), so maybe we can go
>> ahead with that route?
>>
>> - Tim
>>
>> I think it would be good to run the tests on impala.jenkins.io but I
>> don't
>> want to hold up things further. Another interesting technology is the ARM
>> emulation support in Docker.
>>
>> On Sun, Sep 20, 2020 at 8:53 PM Jim Apple <jbap...@apache.org> wrote:
>>
>> > I'm very excited about this development! I don't see any jobs at
>> > status.openlabtesting.org, though -- is
>> >
>> http://status.openlabtesting.org/builds?job_name=impala-build-test-arm64
>> > the right link? Maybe only the owner of a job can see it?
>> >
>> > As far as the price, based on https://ec2instances.info/, you might be
>> > able
>> > to eyeball the cost. I'm no longer a Jenkins admin, so I'll let the
>> other
>> > community members who are admins speak more to this, if it seems
>> > feasible to them.
>> >
>> > On Sat, Sep 19, 2020 at 2:31 AM Tianhua huang <
>> huangtianhua...@gmail.com>
>> > wrote:
>> >
>> > > I am sorry there is no API with Openlab. But we use ZUUL in our
>> system,
>> > you
>> > > can call ZUUL api to get the job information,
>> > > simple like: curl
>> > >
>> > >
>> >
>> http://status.openlabtesting.org/api/builds?job_name=impala-build-test-arm64
>> > > | python -m json.tool
>> > > The result we will get is jobs info of 'impala-build-test-arm64',
>> > including
>> > > job status, job log url and so on, I am not sure if we can integrate
>> with
>> > > the job info into impala.jenkins.io? Then the arm job still runs
>> > triggered
>> > > by OpenLab, and jenkins.impala.io only to get the result and show
>> > > the result in it.
>> > >
>> > > Not sure it's worth doing this, maybe let comm...@impala.apache.org
>> > allow
>> > > receiving the result email from Openlab is more easy:)
>> > >
>> > >
>> > >
>> > > On Sat, Sep 19, 2020 at 11:03 AM Shant Hovsepian <
>> > > sh...@superdupershant.com>
>> > > wrote:
>> > >
>> > > > Would it be possible to setup an API Key or webhook with
>> > openlabstesting?
>> > > >
>> > > > Then we'd have an Impala Jenkins job that uses the API to call
>> > > > openlabstesting to do the build and report the results back.
>> > > >
>> > > > -Shant
>> > > >
>> > > > On Fri, Sep 18, 2020 at 10:24 PM Tianhua huang <
>> > > huangtianhua...@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > Hi Shant,
>> > > > >
>> > > > > Thanks for your reply, maybe it's a good idea to trigger arm
>> testing
>> > by
>> > > > > jenkins.impala.io, we can donate an arm instance, but I don't
>> know
>> > how
>> > > > to
>> > > > > do it. Maybe the Jenkins owner of Impala can help me?
>> > > > >
>> > > > >
>> > > > > On Sat, Sep 19, 2020 at 2:46 AM Shant Hovsepian <
>> > > > sh...@superdupershant.com
>> > > > > >
>> > > > > wrote:
>> > > > >
>> > > > > > Great work Tianhua!
>> > > > > >
>> > > > > > This is a very exciting enhancement.
>> > > > > >
>> > > > > > dev@ isn't usually meant for automated test run output. Would
>> it
>> > be
>> > > > > > possible to get the openlabtesting runs triggered by
>> > > jenkins.impala.io
>> > > > > > then
>> > > > > > have the results of the test run come from that jenkins
>> instance?
>> > > This
>> > > > > > should solve the email problem as the notification can come from
>> > the
>> > > > > > jenkins account and still give us one centralized place to look
>> at
>> > > > > various
>> > > > > > test statuses.
>> > > > > >
>> > > > > > -Shant
>> > > > > >
>> > > > > > On Thu, Sep 17, 2020 at 8:08 AM Tianhua huang <
>> > > > huangtianhua...@gmail.com
>> > > > > >
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi all,
>> > > > > > >
>> > > > > > > Impala can run/test on arm64 platform after our efforts, the
>> > issues
>> > > > we
>> > > > > > > fixed you can find in
>> > > > > https://issues.apache.org/jira/browse/IMPALA-9376
>> > > > > > >
>> > > > > > > We are particularly grateful to Tim and Quanlong for their
>> help,
>> > > > thanks
>> > > > > > > very much!
>> > > > > > >
>> > > > > > > And I have integrated Impala arm64 CI with OpenLab(
>> > > > > > > https://openlabtesting.org/ , a community to do open source
>> > > project
>> > > > > > > testing),
>> > > > > > > see
>> > > > > > >
>> > > > >
>> > >
>> http://status.openlabtesting.org/builds?job_name=impala-build-test-arm64
>> > > > > > > the
>> > > > > > > periodic test job runs for a few days already and runs once
>> per
>> > > day.
>> > > > > > >
>> > > > > > > Tim suggested us to send the arm64 test result to
>> > > > > > > comm...@impala.apache.org,
>> > > > > > > but seems it can receive the emails only ncome from @
>> apache.org.
>> > > > > > > Now I wonder if we can send the test result to Impala
>> community
>> > > > > maillist
>> > > > > > > here dev@impala.apache.org, to tell developers that Impala
>> can
>> > > runs
>> > > > on
>> > > > > > > arm64, to find the compatibility in time.
>> > > > > > >
>> > > > > > > What do you think? Or maybe there is any other good
>> suggestion?
>> > > > > > >
>> > > > > > > Waiting for your reply.
>> > > > > > >
>> > > > > > > Thank you very much!
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
