[
https://issues.apache.org/jira/browse/SPARK-24825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547128#comment-16547128
]
Stavros Kontopoulos edited comment on SPARK-24825 at 7/17/18 9:34 PM:
----------------------------------------------------------------------
As I mentioned elsewhere one thing that worked for me is modify
./dev/make-distribution.sh to do a clean install instead of clean package. But
not sure for the long term for this, although you could purge deps before a
build.
Test suite has the following spark deps:
mvn -f pom.xml dependency:resolve | grep spark
[INFO] org.apache.spark:spark-kvstore_2.11:jar:2.4.0-SNAPSHOT:compile
[INFO] org.apache.spark:spark-network-shuffle_2.11:jar:2.4.0-SNAPSHOT:compile
[INFO] org.apache.spark:spark-unsafe_2.11:jar:2.4.0-SNAPSHOT:compile
[INFO] org.apache.spark:spark-core_2.11:jar:2.4.0-SNAPSHOT:compile
[INFO] org.apache.spark:spark-launcher_2.11:jar:2.4.0-SNAPSHOT:compile
[INFO] org.apache.spark:spark-tags_2.11:jar:2.4.0-SNAPSHOT:compile
[INFO] org.apache.spark:spark-tags_2.11:test-jar:tests:2.4.0-SNAPSHOT:compile
[INFO] org.spark-project.spark:unused:jar:1.0.0:compile
[INFO] org.apache.spark:spark-network-common_2.11:jar:2.4.0-SNAPSHOT:compile
was (Author: skonto):
As I mentioned elsewhere one thing that worked for me is modify
./dev/make-distribution.sh to do a clean install instead of clean package. But
not sure for the long term for this, although you could purge deps.
Test suite has the following spark deps:
mvn -f pom.xml dependency:resolve | grep spark
[INFO] org.apache.spark:spark-kvstore_2.11:jar:2.4.0-SNAPSHOT:compile
[INFO] org.apache.spark:spark-network-shuffle_2.11:jar:2.4.0-SNAPSHOT:compile
[INFO] org.apache.spark:spark-unsafe_2.11:jar:2.4.0-SNAPSHOT:compile
[INFO] org.apache.spark:spark-core_2.11:jar:2.4.0-SNAPSHOT:compile
[INFO] org.apache.spark:spark-launcher_2.11:jar:2.4.0-SNAPSHOT:compile
[INFO] org.apache.spark:spark-tags_2.11:jar:2.4.0-SNAPSHOT:compile
[INFO] org.apache.spark:spark-tags_2.11:test-jar:tests:2.4.0-SNAPSHOT:compile
[INFO] org.spark-project.spark:unused:jar:1.0.0:compile
[INFO] org.apache.spark:spark-network-common_2.11:jar:2.4.0-SNAPSHOT:compile
> [K8S][TEST] Kubernetes integration tests don't trace the maven project
> dependency structure
> -------------------------------------------------------------------------------------------
>
> Key: SPARK-24825
> URL: https://issues.apache.org/jira/browse/SPARK-24825
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes, Tests
> Affects Versions: 2.4.0
> Reporter: Matt Cheah
> Priority: Critical
>
> The Kubernetes integration tests will currently fail if maven installation is
> not performed first, because the integration test build believes it should be
> pulling the Spark parent artifact from maven central. However, this is
> incorrect because the integration test should be building the Spark parent
> pom as a dependency in the multi-module build, and the integration test
> should just use the dynamically built artifact. Or to put it another way, the
> integration test builds should never be pulling Spark dependencies from maven
> central.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
