I am pretty certain you need to change the write.format from “es” to “org.elasticsearch.spark.sql”Sent from my iPhone On 8 Sep 2023, at 03:10, Dipayan Dev <dev.dipaya...@gmail.com> wrote:
++ Dev Hi,
Can you please elaborate your last response? I don’t have any external dependencies added, and just updated the Spark version as mentioned below.
Can someone help me with this? could the provided scope be the issue?
Using the following dependency for Spark 3 in POM file (My Scala version is 2.12.14)
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch-spark-30_2.12</artifactId>
<version>7.12.0</version>
<scope>provided</scope>
</dependency>
The code throws error at this line : df.write.format("es").mode("overwrite").options(elasticOptions).save("index_name") The same code is working with Spark 2.4.0 and the following dependency
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch-spark-20_2.12</artifactId>
<version>7.12.0</version>
</dependency>
What’s the version of the ES connector you are using? Hi All,
We're using Spark 2.4.x to write dataframe into the Elasticsearch index. As we're upgrading to Spark 3.3.0, it throwing out error Caused by: java.lang.ClassNotFoundException: es.DefaultSource
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
Looking at a few responses from Stackoverflow. it seems this is not yet supported by Elasticsearch-hadoop.
Does anyone have experience with this? Or faced/resolved this issue in Spark 3?
Thanks in advance!
Regards
CONFIDENTIALITY NOTICE: This electronic communication and any files transmitted with it are confidential, privileged and intended solely for the use of the individual or entity to whom they are addressed. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution (electronic or otherwise) or forwarding of, or the taking of any action in reliance on the contents of this transmission is strictly prohibited. Please notify the sender immediately by e-mail if you have received this email by mistake and delete this email from your system.
Is it necessary to print this email? If you care about the environment like we do, please refrain from printing emails. It helps to keep the environment forested and litter-free.
|
