Hi all, I am currently attempting to make Iceberg work on the Databricks platform. Without adding IcebergSparkSessionExtensions, I am able to perform both data and metadata reads. However, when I add IcebergSparkSessionExtensions with spark.sql.extensions, Databricks throws the following error:
Caused by: java.lang.NoClassDefFoundError: org/apache/spark/sql/catalyst/analysis/ResolveProcedures ... 41 more Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.catalyst.analysis.ResolveProcedures at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:419) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:352) ... 41 more If I don't add IcebergSparkSessionExtensions, I receive the following error from DBR when I execute UPDATE SQL: Error in SQL statement: DeltaAnalysisException: UPDATE destination only supports Delta sources. The Databricks 12.2 LTS (includes Apache Spark 3.3.2, Scala 2.12) runtime comes with a shaded Iceberg jar with version 0.11. For this testing, I removed it and added iceberg-spark-runtime-3.3_2.12-1.2.1.jar to the classpath. Has anyone successfully used Databricks to perform all Iceberg supported features? Any help would be greatly appreciated. Thank you in advance. Best regards, Pani
