Be interesting to see how well a Pi4 works; with only 4GB of RAM you wouldn't compile with it, but you could try installing the spark jar bundle and then run against some NFS mounted disks: https://www.raspberrypi.org/magpi/raspberry-pi-4-specs-benchmarks/ ; unlikely to be fast, but it'd be an efficient kind of slow
On Fri, Jun 28, 2019 at 3:08 AM Rui Chen <chenrui.m...@gmail.com> wrote: > > I think any AA64 work is going to have to define very clearly what > "works" is defined as > > +1 > It's very valuable to build a clear scope of these projects functionality > for ARM platform in upstream community, it bring confidence to end user and > customers when they plan to deploy these projects on ARM. > > This is absolute long term work, let's to make it step by step, CI, > testing, issue and resolving. > > Steve Loughran <ste...@cloudera.com.invalid> 于2019年6月27日周四 下午9:22写道： > >> level db and native codecs are invariably a problem here, as is anything >> else doing misaligned IO. Protobuf has also had "issues" in the past >> >> see https://issues.apache.org/jira/browse/HADOOP-16100 >> >> I think any AA64 work is going to have to define very clearly what >> "works" is defined as; spark standalone with a specific set of codecs is >> probably the first thing to aim for -no Snappy or lz4. >> >> Anything which goes near: protobuf, checksums, native code, etc is in >> trouble. Don't try and deploy with HDFS as the cluster FS, would be my >> recommendation. >> >> If you want a cluster use NFS or one of google GCS, Azure WASB for the >> cluster FS. And before trying either of those cloud store, run the >> filesystem connector test suites (hadoop-azure; google gcs github) to see >> that they work. If the foundational FS test suites fail, nothing else will >> work >> >> >> >> On Thu, Jun 27, 2019 at 3:09 AM Tianhua huang <huangtianhua...@gmail.com> >> wrote: >> >>> I took the ut tests on my arm instance before and reported an issue in >>> https://issues.apache.org/jira/browse/SPARK-27721, and seems there was >>> no leveldbjni native package for aarch64 in leveldbjni-all.jar(or 1.8) >>> https://mvnrepository.com/artifact/org.fusesource.leveldbjni/leveldbjni-all/1.8 >>> , we can find https://github.com/fusesource/leveldbjni/pull/82 this pr >>> added the aarch64 support and merged on 2 Nov 2017, but the latest release >>> of the repo is on 17 Oct 2013, unfortunately it didn't include the >>> aarch64 supporting. >>> >>> I will running the test on the job mentioned above, and will try to fix >>> the issue above, or if anyone have any idea of it, welcome reply me, thank >>> you. >>> >>> >>> On Wed, Jun 26, 2019 at 8:11 PM Sean Owen <sro...@gmail.com> wrote: >>> >>>> Can you begin by testing yourself? I think the first step is to make >>>> sure the build and tests work on ARM. If you find problems you can >>>> isolate them and try to fix them, or at least report them. It's only >>>> worth getting CI in place when we think builds will work. >>>> >>>> On Tue, Jun 25, 2019 at 9:26 PM Tianhua huang < >>>> huangtianhua...@gmail.com> wrote: >>>> > >>>> > Thanks Shane :) >>>> > >>>> > This sounds good, and yes I agree that it's best to keep the >>>> test/build infrastructure in one place. If you can't find the ARM resource >>>> we are willing to support the ARM instance :) Our goal is to make more >>>> open source software to be more compatible for aarch64 platform, so let's >>>> to do it. I will be happy if I can give some help for the goal. >>>> > >>>> > Waiting for you good news :) >>>> > >>>> > On Wed, Jun 26, 2019 at 9:47 AM shane knapp <skn...@berkeley.edu> >>>> wrote: >>>> >> >>>> >> ...or via VM as you mentioned earlier. :) >>>> >> >>>> >> shane (who will file a JIRA tomorrow) >>>> >> >>>> >> On Tue, Jun 25, 2019 at 6:44 PM shane knapp <skn...@berkeley.edu> >>>> wrote: >>>> >>> >>>> >>> i'd much prefer that we keep the test/build infrastructure in one >>>> place. >>>> >>> >>>> >>> we don't have ARM hardware, but there's a slim possibility i can >>>> scare something up in our older research stock... >>>> >>> >>>> >>> another option would be to run the build in a arm-based docker >>>> container, which (according to the intarwebs) is possible. >>>> >>> >>>> >>> shane >>>> >>> >>>> >>> On Tue, Jun 25, 2019 at 6:35 PM Tianhua huang < >>>> huangtianhua...@gmail.com> wrote: >>>> >>>> >>>> >>>> I forked apache/spark project and propose a job( >>>> https://github.com/theopenlab/spark/pull/1) for spark building in >>>> OpenLab ARM instance, this is the first step to build spark on ARM, I can >>>> enable a periodic job for arm building for apache/spark master if you guys >>>> like. Later I will run tests for spark. I also willing to be the >>>> maintainer of the arm ci of spark. >>>> >>>> >>>> >>>> Thanks for you attention. >>>> >>>> >>>> >>>> On Thu, Jun 20, 2019 at 10:17 AM Tianhua huang < >>>> huangtianhua...@gmail.com> wrote: >>>> >>>>> >>>> >>>>> Thanks Sean. >>>> >>>>> >>>> >>>>> I am very happy to hear that the community will put effort to fix >>>> the ARM-related issues. I'd be happy to help if you like. And could you >>>> give the trace link of this issue, then I can check it is fixed or not, >>>> thank you. >>>> >>>>> As far as I know the old versions of spark support ARM, and now >>>> the new versions don't, this just shows that we need a CI to check whether >>>> the spark support ARM and whether some modification break it. >>>> >>>>> I will add a demo job in OpenLab to build spark on ARM and do a >>>> simple UT test. Later I will give the job link. >>>> >>>>> >>>> >>>>> Let me know what you think. >>>> >>>>> >>>> >>>>> Thank you all! >>>> >>>>> >>>> >>>>> >>>> >>>>> On Wed, Jun 19, 2019 at 8:47 PM Sean Owen <sro...@gmail.com> >>>> wrote: >>>> >>>>>> >>>> >>>>>> I'd begin by reporting and fixing ARM-related issues in the >>>> build. If >>>> >>>>>> they're small, of course we should do them. If it requires >>>> significant >>>> >>>>>> modifications, we can discuss how much Spark can support ARM. I >>>> don't >>>> >>>>>> think it's yet necessary for the Spark project to run these CI >>>> builds >>>> >>>>>> until that point, but it's always welcome if people are testing >>>> that >>>> >>>>>> separately. >>>> >>>>>> >>>> >>>>>> On Wed, Jun 19, 2019 at 7:41 AM Holden Karau < >>>> hol...@pigscanfly.ca> wrote: >>>> >>>>>> > >>>> >>>>>> > Moving to dev@ for increased visibility among the developers. >>>> >>>>>> > >>>> >>>>>> > On Wed, Jun 19, 2019 at 1:24 AM Tianhua huang < >>>> huangtianhua...@gmail.com> wrote: >>>> >>>>>> >> >>>> >>>>>> >> Thanks for your reply. >>>> >>>>>> >> >>>> >>>>>> >> As I said before, I met some problem of build or test for >>>> spark on aarch64 server, so it will be better to have the ARM CI to make >>>> sure the spark is compatible for AArch64 platforms. >>>> >>>>>> >> >>>> >>>>>> >> I’m from OpenLab team(https://openlabtesting.org/ ,a >>>> community to do open source project testing. And we can support some Arm >>>> virtual machines to AMPLab Jenkins, and also we have a developer team that >>>> willing to work on this, we willing to maintain build CI jobs and address >>>> the CI issues. What do you think? >>>> >>>>>> >> >>>> >>>>>> >> >>>> >>>>>> >> Thanks for your attention. >>>> >>>>>> >> >>>> >>>>>> >> >>>> >>>>>> >> On Wed, Jun 19, 2019 at 6:39 AM shane knapp < >>>> skn...@berkeley.edu> wrote: >>>> >>>>>> >>> >>>> >>>>>> >>> yeah, we don't have any aarch64 systems for testing... this >>>> has been asked before but is currently pretty low on our priority list as >>>> we don't have the hardware. >>>> >>>>>> >>> >>>> >>>>>> >>> sorry, >>>> >>>>>> >>> >>>> >>>>>> >>> shane >>>> >>>>>> >>> >>>> >>>>>> >>> On Mon, Jun 10, 2019 at 7:08 PM Tianhua huang < >>>> huangtianhua...@gmail.com> wrote: >>>> >>>>>> >>>> >>>> >>>>>> >>>> Hi, sorry to disturb you. >>>> >>>>>> >>>> The CI testing for apache spark is supported by AMPLab >>>> Jenkins, and I find there are some computers(most of them are Linux (amd64) >>>> arch) for the CI development, but seems there is no Aarch64 computer for >>>> spark CI testing. Recently, I build and run test for spark(master and >>>> branch-2.4) on my arm server, and unfortunately there are some problems, >>>> for example, ut test is failed due to a LEVELDBJNI native package, the >>>> details for java test see http://paste.openstack.org/show/752063/ and >>>> python test see http://paste.openstack.org/show/752709/ >>>> >>>>>> >>>> So I have a question about the ARM CI testing for spark, is >>>> there any plan to support it? Thank you very much and I will wait for your >>>> reply! >>>> >>>>>> >>> >>>> >>>>>> >>> >>>> >>>>>> >>> >>>> >>>>>> >>> -- >>>> >>>>>> >>> Shane Knapp >>>> >>>>>> >>> UC Berkeley EECS Research / RISELab Staff Technical Lead >>>> >>>>>> >>> https://rise.cs.berkeley.edu >>>> >>>>>> > >>>> >>>>>> > >>>> >>>>>> > >>>> >>>>>> > -- >>>> >>>>>> > Twitter: https://twitter.com/holdenkarau >>>> >>>>>> > Books (Learning Spark, High Performance Spark, etc.): >>>> https://amzn.to/2MaRAG9 >>>> >>>>>> > YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>>> >>> >>>> >>> >>>> >>> >>>> >>> -- >>>> >>> Shane Knapp >>>> >>> UC Berkeley EECS Research / RISELab Staff Technical Lead >>>> >>> https://rise.cs.berkeley.edu >>>> >> >>>> >> >>>> >> >>>> >> -- >>>> >> Shane Knapp >>>> >> UC Berkeley EECS Research / RISELab Staff Technical Lead >>>> >> https://rise.cs.berkeley.edu >>>> >>>
