++ Dev
On Thu, 7 Sep 2023 at 10:22 PM, Dipayan Dev <dev.dipaya...@gmail.com> wrote:
> Hi,
>
> Can you please elaborate your last response? I don’t have any external
> dependencies added, and just updated the Spark version as mentioned below.
>
> Can someone help me with this?
>
> On Fri, 1 Sep 2023 at 5:58 PM, Koert Kuipers <ko...@tresata.com> wrote:
>
>> could the provided scope be the issue?
>>
>> On Sun, Aug 27, 2023 at 2:58 PM Dipayan Dev <dev.dipaya...@gmail.com>
>> wrote:
>>
>>> Using the following dependency for Spark 3 in POM file (My Scala version
>>> is 2.12.14)
>>>
>>>
>>>
>>>
>>>
>>>
>>> *<dependency> <groupId>org.elasticsearch</groupId>
>>> <artifactId>elasticsearch-spark-30_2.12</artifactId>
>>> <version>7.12.0</version> <scope>provided</scope></dependency>*
>>>
>>>
>>> The code throws error at this line :
>>> df.write.format("es").mode("overwrite").options(elasticOptions).save("index_name")
>>> The same code is working with Spark 2.4.0 and the following dependency
>>>
>>>
>>>
>>>
>>>
>>> *<dependency> <groupId>org.elasticsearch</groupId>
>>> <artifactId>elasticsearch-spark-20_2.12</artifactId>
>>> <version>7.12.0</version></dependency>*
>>>
>>>
>>> On Mon, 28 Aug 2023 at 12:17 AM, Holden Karau <hol...@pigscanfly.ca>
>>> wrote:
>>>
>>>> What’s the version of the ES connector you are using?
>>>>
>>>> On Sat, Aug 26, 2023 at 10:17 AM Dipayan Dev <dev.dipaya...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi All,
>>>>>
>>>>> We're using Spark 2.4.x to write dataframe into the Elasticsearch
>>>>> index.
>>>>> As we're upgrading to Spark 3.3.0, it throwing out error
>>>>> Caused by: java.lang.ClassNotFoundException: es.DefaultSource
>>>>> at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
>>>>> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
>>>>> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
>>>>>
>>>>> Looking at a few responses from Stackoverflow
>>>>> <https://stackoverflow.com/a/66452149>. it seems this is not yet
>>>>> supported by Elasticsearch-hadoop.
>>>>>
>>>>> Does anyone have experience with this? Or faced/resolved this issue in
>>>>> Spark 3?
>>>>>
>>>>> Thanks in advance!
>>>>>
>>>>> Regards
>>>>> Dipayan
>>>>>
>>>> --
>>>> Twitter: https://twitter.com/holdenkarau
>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>>
>>>
>> CONFIDENTIALITY NOTICE: This electronic communication and any files
>> transmitted with it are confidential, privileged and intended solely for
>> the use of the individual or entity to whom they are addressed. If you are
>> not the intended recipient, you are hereby notified that any disclosure,
>> copying, distribution (electronic or otherwise) or forwarding of, or the
>> taking of any action in reliance on the contents of this transmission is
>> strictly prohibited. Please notify the sender immediately by e-mail if you
>> have received this email by mistake and delete this email from your system.
>>
>> Is it necessary to print this email? If you care about the environment
>> like we do, please refrain from printing emails. It helps to keep the
>> environment forested and litter-free.
>
>
