[
https://issues.apache.org/jira/browse/SPARK-29106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17188217#comment-17188217
]
huangtianhua edited comment on SPARK-29106 at 9/1/20, 7:28 AM:
---------------------------------------------------------------
[~shaneknapp] and [~srowen], sorry to disturb you. There is an issue for spark
arm64 jenkins, the test 'org.apache.spark.DistributedSuite' failed for several
days, and I don't know the reason and can't fix it, see
https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-arm, and I
have filed a jira SPARK-32691, could you have a look for it, I can provide
arm64 instance if needed, thanks very much.
ps: seems the issue happended after the commit
b421bf0196897fa66d6d15ba6461a24b23ac6dd3
was (Author: huangtianhua):
[~shaneknapp] and [~srowen], sorry to disturb you. There is an issue for spark
arm64 jenkins, the test 'org.apache.spark.DistributedSuite' failed for several
days, and I don't know the reason and can't fix it, see
https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-arm, and I
have filed a jira for SPARK-32691, could you have a look for it, I can provide
arm64 instance if needed, thanks very much.
> Add jenkins arm test for spark
> ------------------------------
>
> Key: SPARK-29106
> URL: https://issues.apache.org/jira/browse/SPARK-29106
> Project: Spark
> Issue Type: Test
> Components: Tests
> Affects Versions: 3.0.0
> Reporter: huangtianhua
> Assignee: Shane Knapp
> Priority: Minor
> Attachments: R-ansible.yml, R-libs.txt,
> SparkR-and-pyspark36-testing.txt, arm-python36.txt
>
>
> Add arm test jobs to amplab jenkins for spark.
> Till now we made two arm test periodic jobs for spark in OpenLab, one is
> based on master with hadoop 2.7(similar with QA test of amplab jenkins),
> other one is based on a new branch which we made on date 09-09, see
> [http://status.openlabtesting.org/builds/job/spark-master-unit-test-hadoop-2.7-arm64]
> and
> [http://status.openlabtesting.org/builds/job/spark-unchanged-branch-unit-test-hadoop-2.7-arm64.|http://status.openlabtesting.org/builds/job/spark-unchanged-branch-unit-test-hadoop-2.7-arm64]
> We only have to care about the first one when integrate arm test with amplab
> jenkins.
> About the k8s test on arm, we have took test it, see
> [https://github.com/theopenlab/spark/pull/17], maybe we can integrate it
> later.
> And we plan test on other stable branches too, and we can integrate them to
> amplab when they are ready.
> We have offered an arm instance and sent the infos to shane knapp, thanks
> shane to add the first arm job to amplab jenkins :)
> The other important thing is about the leveldbjni
> [https://github.com/fusesource/leveldbjni,|https://github.com/fusesource/leveldbjni/issues/80]
> spark depends on leveldbjni-all-1.8
> [https://mvnrepository.com/artifact/org.fusesource.leveldbjni/leveldbjni-all/1.8],
> we can see there is no arm64 supporting. So we build an arm64 supporting
> release of leveldbjni see
> [https://mvnrepository.com/artifact/org.openlabtesting.leveldbjni/leveldbjni-all/1.8],
> but we can't modified the spark pom.xml directly with something like
> 'property'/'profile' to choose correct jar package on arm or x86 platform,
> because spark depends on some hadoop packages like hadoop-hdfs, the packages
> depend on leveldbjni-all-1.8 too, unless hadoop release with new arm
> supporting leveldbjni jar. Now we download the leveldbjni-al-1.8 of
> openlabtesting and 'mvn install' to use it when arm testing for spark.
> PS: The issues found and fixed:
> SPARK-28770
> [https://github.com/apache/spark/pull/25673]
>
> SPARK-28519
> [https://github.com/apache/spark/pull/25279]
>
> SPARK-28433
> [https://github.com/apache/spark/pull/25186]
>
> SPARK-28467
> [https://github.com/apache/spark/pull/25864]
>
> SPARK-29286
> [https://github.com/apache/spark/pull/26021]
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
