[
https://issues.apache.org/jira/browse/SPARK-24825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547061#comment-16547061
]
Stavros Kontopoulos edited comment on SPARK-24825 at 7/17/18 8:54 PM:
----------------------------------------------------------------------
To reproduce it try build the test suite:
../../../build/mvn install
Using `mvn` from path:...spark/build/apache-maven-3.3.9/bin/mvn
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Spark Project Kubernetes Integration Tests 2.4.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
Downloading:
[https://repository.apache.org/snapshots/org/apache/spark/spark-core_2.11/2.4.0-SNAPSHOT/maven-metadata.xml]
Downloaded:
[https://repository.apache.org/snapshots/org/apache/spark/spark-core_2.11/2.4.0-SNAPSHOT/maven-metadata.xml]
(2 KB at 0.4 KB/sec)
Downloading:
[https://repository.apache.org/snapshots/org/apache/spark/spark-core_2.11/2.4.0-SNAPSHOT/spark-core_2.11-2.4.0-20180717.095738-170.pom]
[WARNING] The POM for
org.apache.spark:spark-core_2.11:jar:2.4.0-20180717.095738-170 is missing, no
dependency information available
[WARNING] The POM for
org.apache.spark:spark-core_2.11:jar:tests:2.4.0-20180717.095737-170 is
missing, no dependency information available
Downloading:
[https://repository.apache.org/snapshots/org/apache/spark/spark-tags_2.11/2.4.0-SNAPSHOT/maven-metadata.xml]
Downloaded:
[https://repository.apache.org/snapshots/org/apache/spark/spark-tags_2.11/2.4.0-SNAPSHOT/maven-metadata.xml]
(2 KB at 0.7 KB/sec)
Downloading:
[https://repository.apache.org/snapshots/org/apache/spark/spark-tags_2.11/2.4.0-SNAPSHOT/spark-tags_2.11-2.4.0-20180717.095221-170.pom]
[WARNING] The POM for
org.apache.spark:spark-tags_2.11:jar:tests:2.4.0-20180717.095220-170 is
missing, no dependency information available
Downloading:
[https://repository.apache.org/snapshots/org/apache/spark/spark-core_2.11/2.4.0-SNAPSHOT/spark-core_2.11-2.4.0-20180717.095738-170.jar]
Downloading:
[https://repository.apache.org/snapshots/org/apache/spark/spark-core_2.11/2.4.0-SNAPSHOT/spark-core_2.11-2.4.0-20180717.095737-170-tests.jar]
Downloading:
[https://repository.apache.org/snapshots/org/apache/spark/spark-tags_2.11/2.4.0-SNAPSHOT/spark-tags_2.11-2.4.0-20180717.095220-170-tests.jar]
Downloaded:
[https://repository.apache.org/snapshots/org/apache/spark/spark-tags_2.11/2.4.0-SNAPSHOT/spark-tags_2.11-2.4.0-20180717.095220-170-tests.jar]
(9 KB at 5.2 KB/sec)
Downloaded:
[https://repository.apache.org/snapshots/org/apache/spark/spark-core_2.11/2.4.0-SNAPSHOT/spark-core_2.11-2.4.0-20180717.095737-170-tests.jar]
(10438 KB at 577.5 KB/sec)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 26.346 s
[INFO] Finished at: 2018-07-17T23:15:29+03:00
[INFO] Final Memory: 14M/204M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project
spark-kubernetes-integration-tests_2.11: Could not resolve dependencies for
project
spark-kubernetes-integration-tests:spark-kubernetes-integration-tests_2.11:jar:2.4.0-SNAPSHOT:
Could not find artifact
org.apache.spark:spark-core_2.11:jar:2.4.0-20180717.095738-170 in
apache.snapshots ([https://repository.apache.org/snapshots]) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
[http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionExceptio|http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException]
The metadata here show expected version:
[https://repository.apache.org/content/groups/snapshots/org/apache/spark/spark-core_2.11/2.4.0-SNAPSHOT/maven-metadata.xml]
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.4.0-SNAPSHOT</version>
<versioning>
<snapshot>
<timestamp>20180717.095738</timestamp>
<buildNumber>170</buildNumber>
</snapshot>
<lastUpdated>20180717095738</lastUpdated>
but there is no such jar uploaded.
Latest is:
[spark-core_2.11-2.4.0-20180717.095737-170.jar|https://repository.apache.org/content/groups/snapshots/org/apache/spark/spark-core_2.11/2.4.0-SNAPSHOT/spark-core_2.11-2.4.0-20180717.095737-170.jar]
Probably timestamp is wrong here:
was (Author: skonto):
To reproduce it try build the test suite:
../../../build/mvn install
Using `mvn` from path:...spark/build/apache-maven-3.3.9/bin/mvn
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Spark Project Kubernetes Integration Tests 2.4.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
Downloading:
[https://repository.apache.org/snapshots/org/apache/spark/spark-core_2.11/2.4.0-SNAPSHOT/maven-metadata.xml]
Downloaded:
[https://repository.apache.org/snapshots/org/apache/spark/spark-core_2.11/2.4.0-SNAPSHOT/maven-metadata.xml]
(2 KB at 0.4 KB/sec)
Downloading:
[https://repository.apache.org/snapshots/org/apache/spark/spark-core_2.11/2.4.0-SNAPSHOT/spark-core_2.11-2.4.0-20180717.095738-170.pom]
[WARNING] The POM for
org.apache.spark:spark-core_2.11:jar:2.4.0-20180717.095738-170 is missing, no
dependency information available
[WARNING] The POM for
org.apache.spark:spark-core_2.11:jar:tests:2.4.0-20180717.095737-170 is
missing, no dependency information available
Downloading:
[https://repository.apache.org/snapshots/org/apache/spark/spark-tags_2.11/2.4.0-SNAPSHOT/maven-metadata.xml]
Downloaded:
[https://repository.apache.org/snapshots/org/apache/spark/spark-tags_2.11/2.4.0-SNAPSHOT/maven-metadata.xml]
(2 KB at 0.7 KB/sec)
Downloading:
[https://repository.apache.org/snapshots/org/apache/spark/spark-tags_2.11/2.4.0-SNAPSHOT/spark-tags_2.11-2.4.0-20180717.095221-170.pom]
[WARNING] The POM for
org.apache.spark:spark-tags_2.11:jar:tests:2.4.0-20180717.095220-170 is
missing, no dependency information available
Downloading:
[https://repository.apache.org/snapshots/org/apache/spark/spark-core_2.11/2.4.0-SNAPSHOT/spark-core_2.11-2.4.0-20180717.095738-170.jar]
Downloading:
[https://repository.apache.org/snapshots/org/apache/spark/spark-core_2.11/2.4.0-SNAPSHOT/spark-core_2.11-2.4.0-20180717.095737-170-tests.jar]
Downloading:
[https://repository.apache.org/snapshots/org/apache/spark/spark-tags_2.11/2.4.0-SNAPSHOT/spark-tags_2.11-2.4.0-20180717.095220-170-tests.jar]
Downloaded:
[https://repository.apache.org/snapshots/org/apache/spark/spark-tags_2.11/2.4.0-SNAPSHOT/spark-tags_2.11-2.4.0-20180717.095220-170-tests.jar]
(9 KB at 5.2 KB/sec)
Downloaded:
[https://repository.apache.org/snapshots/org/apache/spark/spark-core_2.11/2.4.0-SNAPSHOT/spark-core_2.11-2.4.0-20180717.095737-170-tests.jar]
(10438 KB at 577.5 KB/sec)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 26.346 s
[INFO] Finished at: 2018-07-17T23:15:29+03:00
[INFO] Final Memory: 14M/204M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project
spark-kubernetes-integration-tests_2.11: Could not resolve dependencies for
project
spark-kubernetes-integration-tests:spark-kubernetes-integration-tests_2.11:jar:2.4.0-SNAPSHOT:
Could not find artifact
org.apache.spark:spark-core_2.11:jar:2.4.0-20180717.095738-170 in
apache.snapshots ([https://repository.apache.org/snapshots]) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
[http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionExceptio|http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException]
The metadata here show expected version:
[https://repository.apache.org/content/groups/snapshots/org/apache/spark/spark-core_2.11/2.4.0-SNAPSHOT/maven-metadata.xml]
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.4.0-SNAPSHOT</version>
<versioning>
<snapshot>
<timestamp>20180717.095738</timestamp>
<buildNumber>170</buildNumber>
</snapshot>
<lastUpdated>20180717095738</lastUpdated>
but there is no such jar uploaded.
> [K8S][TEST] Kubernetes integration tests don't trace the maven project
> dependency structure
> -------------------------------------------------------------------------------------------
>
> Key: SPARK-24825
> URL: https://issues.apache.org/jira/browse/SPARK-24825
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes, Tests
> Affects Versions: 2.4.0
> Reporter: Matt Cheah
> Priority: Critical
>
> The Kubernetes integration tests will currently fail if maven installation is
> not performed first, because the integration test build believes it should be
> pulling the Spark parent artifact from maven central. However, this is
> incorrect because the integration test should be building the Spark parent
> pom as a dependency in the multi-module build, and the integration test
> should just use the dynamically built artifact. Or to put it another way, the
> integration test builds should never be pulling Spark dependencies from maven
> central.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
