[
https://issues.apache.org/jira/browse/SPARK-29106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16957603#comment-16957603
]
zhao bo commented on SPARK-29106:
---------------------------------
Hi [~shaneknapp],
Sorry for disturb. I have some questions about the following work want to
discuss with you. I list them in the following.
# For pyspark test, you mentioned we didn't install any python debs for
testing. Is there any "requirements.txt" or "test-requirements.txt" in the
spark repo? I'm failed to find them. When we test the pyspark before, we just
realize that we need to install numpy package with pip, because when we exec
the pyspark test scripts, the fail message noticed us. So you mentioned
"pyspark testing debs" before, you mean that we should figure all out manually?
Is there any kind suggest from your side?
# For sparkR test, we compile a higher R version 3.6.1 by fix many lib
dependency, and make it work. And exec the R test script, till to all of them
return pass. So we wonder the difficult about the test when we truelly in
amplab, could you please share more to us?
# For current periodic jobs, you said it will be triggered 2 times per day.
Each build will cost most 11 hours. I have a thought about the next job
deployment, wish to know your thought about it. My thought is we can setup 2
jobs per day, one is the current maven UT test triggered by SCM changes(11h),
the other will run the pyspark and sparkR tests also triggered by SCM
changes(including spark build and tests, may cost 5-6 hours). How about this?
We can talk and discuss if we don't realize how difficult to do these now.
Thanks very much, shane. And hope you could reply if you are free. ;)
> Add jenkins arm test for spark
> ------------------------------
>
> Key: SPARK-29106
> URL: https://issues.apache.org/jira/browse/SPARK-29106
> Project: Spark
> Issue Type: Test
> Components: Tests
> Affects Versions: 3.0.0
> Reporter: huangtianhua
> Priority: Minor
>
> Add arm test jobs to amplab jenkins for spark.
> Till now we made two arm test periodic jobs for spark in OpenLab, one is
> based on master with hadoop 2.7(similar with QA test of amplab jenkins),
> other one is based on a new branch which we made on date 09-09, see
> [http://status.openlabtesting.org/builds/job/spark-master-unit-test-hadoop-2.7-arm64]
> and
> [http://status.openlabtesting.org/builds/job/spark-unchanged-branch-unit-test-hadoop-2.7-arm64.|http://status.openlabtesting.org/builds/job/spark-unchanged-branch-unit-test-hadoop-2.7-arm64]
> We only have to care about the first one when integrate arm test with amplab
> jenkins.
> About the k8s test on arm, we have took test it, see
> [https://github.com/theopenlab/spark/pull/17], maybe we can integrate it
> later.
> And we plan test on other stable branches too, and we can integrate them to
> amplab when they are ready.
> We have offered an arm instance and sent the infos to shane knapp, thanks
> shane to add the first arm job to amplab jenkins :)
> The other important thing is about the leveldbjni
> [https://github.com/fusesource/leveldbjni,|https://github.com/fusesource/leveldbjni/issues/80]
> spark depends on leveldbjni-all-1.8
> [https://mvnrepository.com/artifact/org.fusesource.leveldbjni/leveldbjni-all/1.8],
> we can see there is no arm64 supporting. So we build an arm64 supporting
> release of leveldbjni see
> [https://mvnrepository.com/artifact/org.openlabtesting.leveldbjni/leveldbjni-all/1.8],
> but we can't modified the spark pom.xml directly with something like
> 'property'/'profile' to choose correct jar package on arm or x86 platform,
> because spark depends on some hadoop packages like hadoop-hdfs, the packages
> depend on leveldbjni-all-1.8 too, unless hadoop release with new arm
> supporting leveldbjni jar. Now we download the leveldbjni-al-1.8 of
> openlabtesting and 'mvn install' to use it when arm testing for spark.
> PS: The issues found and fixed:
> SPARK-28770
> [https://github.com/apache/spark/pull/25673]
>
> SPARK-28519
> [https://github.com/apache/spark/pull/25279]
>
> SPARK-28433
> [https://github.com/apache/spark/pull/25186]
>
> SPARK-28467
> [https://github.com/apache/spark/pull/25864]
>
> SPARK-29286
> [https://github.com/apache/spark/pull/26021]
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
