Hi John,
I tried using both the "set spark.sql.extensions" command and
"spark.conf.get("spark.sql.extensions")" method, but both returned
"undefined."To resolve this issue, I attempted to set the following values: spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension,org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions However, these attempts were unsuccessful. I'm not sure if Databricks allows users to override the spark.sql.extensions configuration. Thank you for your help. Thanks Pani On Fri, Apr 28, 2023 at 3:04 PM John Zhuge <jzh...@apache.org> wrote: > Have you tried this? > > > 1. Copy the value of `spark.sql.extensions` (a comma separated > list) from an existing Spark job on Databricks > 2. Append > `org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions` to the > comma separated list > 3. Submit a new job with `spark.sql.extensions` set to the modified > value > > > On Fri, Apr 28, 2023 at 12:28 PM Pani Dhakshnamurthy <dpa...@gmail.com> > wrote: > >> Hi all, >> >> I am currently attempting to make Iceberg work on the Databricks >> platform. Without adding IcebergSparkSessionExtensions, I am able to >> perform both data and metadata reads. However, when I add >> IcebergSparkSessionExtensions with spark.sql.extensions, Databricks throws >> the following error: >> >> Caused by: java.lang.NoClassDefFoundError: >> org/apache/spark/sql/catalyst/analysis/ResolveProcedures >> ... 41 more >> Caused by: java.lang.ClassNotFoundException: >> org.apache.spark.sql.catalyst.analysis.ResolveProcedures >> at java.net.URLClassLoader.findClass(URLClassLoader.java:387) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:419) >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:352) >> ... 41 more >> >> If I don't add IcebergSparkSessionExtensions, I receive the following >> error from DBR when I execute UPDATE SQL: Error in SQL statement: >> DeltaAnalysisException: UPDATE destination only supports Delta sources. >> >> The Databricks 12.2 LTS (includes Apache Spark 3.3.2, Scala 2.12) runtime >> comes with a shaded Iceberg jar with version 0.11. For this testing, I >> removed it and added iceberg-spark-runtime-3.3_2.12-1.2.1.jar to the >> classpath. >> >> Has anyone successfully used Databricks to perform all Iceberg supported >> features? Any help would be greatly appreciated. Thank you in advance. >> >> Best regards, >> Pani >> > > > -- > John Zhuge >
