wood-dunamu opened a new issue, #8544:
URL: https://github.com/apache/iceberg/issues/8544
### Apache Iceberg version
1.3.1 (latest release)
### Query engine
EMR
### Please describe the bug 🐞
### Summary
- Facing warning when starting spark-sql in EMR using Glue Catalog
- Just warning, not error. So, It's possible to run spark sql, but I
want to know how can I remove that warning.
- Spark version: 3.4.0
- EMR version: 6.12.0
### Spark command
```bash
spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-3.4_2.12:1.3.0
\
--conf
spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog \
--conf spark.sql.catalog.spark_catalog.type=hive \
--conf spark.sql.catalog.iceberg=org.apache.iceberg.spark.SparkCatalog \
--conf spark.sql.catalog.iceberg.warehouse=s3://test-bucket/iceberg/ \
--conf
spark.sql.catalog.iceberg.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog \
--conf
spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
\
--conf
spark.sql.catalog.iceberg.io-impl=org.apache.iceberg.aws.s3.S3FileIO
```
### Error stacktrace
```bash
23/09/10 13:48:50 WARN IcebergBuild: Failed to load version properties from
/iceberg-build.properties
java.lang.IllegalArgumentException: resource /iceberg-build.properties
relative to org.apache.iceberg.IcebergBuild not found.
at
org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkArgument(Preconditions.java:435)
~[iceberg-spark-runtime-3.4_2.12-1.3.0-amzn-0.jar:?]
at
org.apache.iceberg.relocated.com.google.common.io.Resources.getResource(Resources.java:207)
~[iceberg-spark-runtime-3.4_2.12-1.3.0-amzn-0.jar:?]
at org.apache.iceberg.IcebergBuild.readResource(IcebergBuild.java:112)
~[iceberg-spark-runtime-3.4_2.12-1.3.0-amzn-0.jar:?]
at org.apache.iceberg.IcebergBuild.loadBuildInfo(IcebergBuild.java:51)
~[iceberg-spark-runtime-3.4_2.12-1.3.0-amzn-0.jar:?]
at org.apache.iceberg.IcebergBuild.ensureLoaded(IcebergBuild.java:104)
~[iceberg-spark-runtime-3.4_2.12-1.3.0-amzn-0.jar:?]
at org.apache.iceberg.IcebergBuild.fullVersion(IcebergBuild.java:96)
~[iceberg-spark-runtime-3.4_2.12-1.3.0-amzn-0.jar:?]
at
org.apache.iceberg.EnvironmentContext.<clinit>(EnvironmentContext.java:34)
~[iceberg-spark-runtime-3.4_2.12-1.3.0-amzn-0.jar:?]
at
org.apache.iceberg.spark.SparkCatalog.initialize(SparkCatalog.java:571)
~[iceberg-spark-runtime-3.4_2.12-1.3.0-amzn-0.jar:?]
at
org.apache.spark.sql.connector.catalog.Catalogs$.load(Catalogs.scala:65)
~[spark-catalyst_2.12-3.4.0-amzn-0.jar:3.4.0-amzn-0]
at
org.apache.spark.sql.connector.catalog.CatalogManager.$anonfun$catalog$1(CatalogManager.scala:53)
~[spark-catalyst_2.12-3.4.0-amzn-0.jar:3.4.0-amzn-0]
at scala.collection.mutable.HashMap.getOrElseUpdate(HashMap.scala:86)
~[scala-library-2.12.15.jar:?]
at
org.apache.spark.sql.connector.catalog.CatalogManager.catalog(CatalogManager.scala:53)
~[spark-catalyst_2.12-3.4.0-amzn-0.jar:3.4.0-amzn-0]
at
org.apache.spark.sql.connector.catalog.CatalogManager.currentCatalog(CatalogManager.scala:122)
~[spark-catalyst_2.12-3.4.0-amzn-0.jar:3.4.0-amzn-0]
at
org.apache.spark.sql.connector.catalog.CatalogManager.currentNamespace(CatalogManager.scala:93)
~[spark-catalyst_2.12-3.4.0-amzn-0.jar:3.4.0-amzn-0]
at
org.apache.spark.sql.internal.CatalogImpl.currentDatabase(CatalogImpl.scala:65)
~[spark-sql_2.12-3.4.0-amzn-0.jar:3.4.0-amzn-0]
at
org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.currentDB$1(SparkSQLCLIDriver.scala:285)
~[spark-hive-thriftserver_2.12-3.4.0-amzn-0.jar:3.4.0-amzn-0]
at
org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.promptWithCurrentDB$1(SparkSQLCLIDriver.scala:292)
~[spark-hive-thriftserver_2.12-3.4.0-amzn-0.jar:3.4.0-amzn-0]
at
org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:296)
~[spark-hive-thriftserver_2.12-3.4.0-amzn-0.jar:3.4.0-amzn-0]
at
org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
~[spark-hive-thriftserver_2.12-3.4.0-amzn-0.jar:3.4.0-amzn-0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_382]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_382]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_382]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_382]
at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
~[spark-core_2.12-3.4.0-amzn-0.jar:3.4.0-amzn-0]
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1066)
~[spark-core_2.12-3.4.0-amzn-0.jar:3.4.0-amzn-0]
at
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
~[spark-core_2.12-3.4.0-amzn-0.jar:3.4.0-amzn-0]
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
~[spark-core_2.12-3.4.0-amzn-0.jar:3.4.0-amzn-0]
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
~[spark-core_2.12-3.4.0-amzn-0.jar:3.4.0-amzn-0]
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1158)
~[spark-core_2.12-3.4.0-amzn-0.jar:3.4.0-amzn-0]
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1167)
~[spark-core_2.12-3.4.0-amzn-0.jar:3.4.0-amzn-0]
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
~[spark-core_2.12-3.4.0-amzn-0.jar:3.4.0-amzn-0]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
