It looks like you are still pulling in some 0.11 version classes. That version is very old and may not contain `ResolveProcedures`. I assume you have classpath first flag enabled?
- Anton > On Apr 28, 2023, at 3:18 PM, Pani Dhakshnamurthy <[email protected]> wrote: > > Hi John, > I tried using both the "set spark.sql.extensions" command and > "spark.conf.get("spark.sql.extensions")" method, but both returned > "undefined." > > To resolve this issue, I attempted to set the following values: > > spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions > > spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension,org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions > > However, these attempts were unsuccessful. > > I'm not sure if Databricks allows users to override the spark.sql.extensions > configuration. > > Thank you for your help. > > Thanks > Pani > > > On Fri, Apr 28, 2023 at 3:04 PM John Zhuge <[email protected] > <mailto:[email protected]>> wrote: > Have you tried this? > > Copy the value of `spark.sql.extensions` (a comma separated list) from an > existing Spark job on Databricks > Append `org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions` > to the comma separated list > Submit a new job with `spark.sql.extensions` set to the modified value > > On Fri, Apr 28, 2023 at 12:28 PM Pani Dhakshnamurthy <[email protected] > <mailto:[email protected]>> wrote: > Hi all, > > I am currently attempting to make Iceberg work on the Databricks platform. > Without adding IcebergSparkSessionExtensions, I am able to perform both data > and metadata reads. However, when I add IcebergSparkSessionExtensions with > spark.sql.extensions, Databricks throws the following error: > > Caused by: java.lang.NoClassDefFoundError: > org/apache/spark/sql/catalyst/analysis/ResolveProcedures > ... 41 more > Caused by: java.lang.ClassNotFoundException: > org.apache.spark.sql.catalyst.analysis.ResolveProcedures > at java.net.URLClassLoader.findClass(URLClassLoader.java:387) > at java.lang.ClassLoader.loadClass(ClassLoader.java:419) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) > at java.lang.ClassLoader.loadClass(ClassLoader.java:352) > ... 41 more > > If I don't add IcebergSparkSessionExtensions, I receive the following error > from DBR when I execute UPDATE SQL: Error in SQL statement: > DeltaAnalysisException: UPDATE destination only supports Delta sources. > > The Databricks 12.2 LTS (includes Apache Spark 3.3.2, Scala 2.12) runtime > comes with a shaded Iceberg jar with version 0.11. For this testing, I > removed it and added iceberg-spark-runtime-3.3_2.12-1.2.1.jar to the > classpath. > > Has anyone successfully used Databricks to perform all Iceberg supported > features? Any help would be greatly appreciated. Thank you in advance. > > Best regards, > Pani > > > -- > John Zhuge
