racevedoo opened a new issue #3620: URL: https://github.com/apache/iceberg/issues/3620
Hi all! I'm trying to use iceberg's snapshot release (to make use of the better spark 3.2 support), but I'm getting the following error when running spark-shell (full log [here](https://gist.github.com/racevedoo/3530bce04364629f81217e78dea66071)) ``` spark-shell --repositories https://repository.apache.org/content/repositories/snapshots \ --packages org.apache.iceberg:iceberg-spark-3.2-runtime:0.13.0-SNAPSHOT,org.postgresql:postgresql:42.2.10,software.amazon.awssdk:url-connection-client:2.17.68,software.amazon.awssdk:sts:2.17.68,software.amazon.awssdk:s3:2.17.68,software.amazon.awssdk:dynamodb:2.17.68,software.amazon.awssdk:glue:2.17.68 :: problems summary :: :::: WARNINGS module not found: org.slf4j#slf4j-api;working@mymachine ==== local-m2-cache: tried file:/home/rafael.acevedo/.m2/repository/org/slf4j/slf4j-api/working@mymachine/[email protected] -- artifact org.slf4j#slf4j-api;working@mymachine!slf4j-api.jar: file:/home/rafael.acevedo/.m2/repository/org/slf4j/slf4j-api/working@mymachine/[email protected] ==== local-ivy-cache: tried /home/rafael.acevedo/.ivy2/local/org.slf4j/slf4j-api/working@mymachine/ivys/ivy.xml -- artifact org.slf4j#slf4j-api;working@mymachine!slf4j-api.jar: /home/rafael.acevedo/.ivy2/local/org.slf4j/slf4j-api/working@mymachine/jars/slf4j-api.jar ==== central: tried https://repo1.maven.org/maven2/org/slf4j/slf4j-api/working@mymachine/[email protected] -- artifact org.slf4j#slf4j-api;working@mymachine!slf4j-api.jar: https://repo1.maven.org/maven2/org/slf4j/slf4j-api/working@mymachine/[email protected] ==== spark-packages: tried https://repos.spark-packages.org/org/slf4j/slf4j-api/working@mymachine/[email protected] -- artifact org.slf4j#slf4j-api;working@mymachine!slf4j-api.jar: https://repos.spark-packages.org/org/slf4j/slf4j-api/working@mymachine/[email protected] ==== repo-1: tried https://repository.apache.org/content/repositories/snapshots/org/slf4j/slf4j-api/working@mymachine/[email protected] -- artifact org.slf4j#slf4j-api;working@mymachine!slf4j-api.jar: https://repository.apache.org/content/repositories/snapshots/org/slf4j/slf4j-api/working@mymachine/[email protected] :::::::::::::::::::::::::::::::::::::::::::::: :: UNRESOLVED DEPENDENCIES :: :::::::::::::::::::::::::::::::::::::::::::::: :: org.slf4j#slf4j-api;working@mymachine: not found :::::::::::::::::::::::::::::::::::::::::::::: :::: ERRORS unknown resolver null unknown resolver null unknown resolver public unknown resolver null unknown resolver null unknown resolver null unknown resolver null unknown resolver null unknown resolver null unknown resolver null unknown resolver null unknown resolver null unknown resolver null unknown resolver null unknown resolver null unknown resolver null unknown resolver null unknown resolver sbt-chain unknown resolver null unknown resolver sbt-chain unknown resolver null unknown resolver null unknown resolver null unknown resolver null unknown resolver null unknown resolver null unknown resolver null unknown resolver null unknown resolver null unknown resolver null :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS Exception in thread "main" java.lang.RuntimeException: [unresolved dependency: org.slf4j#slf4j-api;working@mymachine: not found] at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1447) at org.apache.spark.util.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:185) at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:308) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:898) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) ``` Running with iceberg 0.12.1 works fine: ``` spark-shell --repositories https://repository.apache.org/content/repositories/snapshots --packages org.apache.iceberg:iceberg-spark3-runtime:0.12.1,org.postgresql:postgresql:42.2.10,software.amazon.awssdk:url-connection-client:2.17.68,software.amazon.awssdk:sts:2.17.68,software.amazon.awssdk:s3:2.17.68,software.amazon.awssdk:dynamodb:2.17.68,software.amazon.awssdk:glue:2.17.68 ``` From my investigation, slf4j is now a runtime dependency (should be included with spark) since #3335 (https://github.com/apache/iceberg/blob/master/spark/v3.2/build.gradle#L181). In my case, dependency resolution is failing for some reason -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
