[ 
https://issues.apache.org/jira/browse/IGNITE-12054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927325#comment-16927325
 ] 

Nikolay Izhikov commented on IGNITE-12054:
------------------------------------------

Hello, [~zaleslaw]

The main issue with upgrading to 2.4 and support both 2.3 and 2.5 is changes in 
Spark internal API that was introduced in 2.3 version.
As you may know, Ignite - Spark integration uses some internal API 
({{ExternalCatalog}}) and leverage on other parts of close-to-internal parts of 
Spark(query parser and optimizer). That code was changed in 2.4 and we can't 
simply change the version in pom.xml. We need to fix our code for the new Spark 
behaviour.

I tried to do so but fail to do it quickly.
If you want to take care of this upgrade, please, do.

As for support both version, it seems we should have different modules or 
similar to do so.

[~dmagda] Ignite, in the past, had those kinds of modules, that support two 
different versions of some external library.
Should we do the similar and create spark_24 and spark_25 modules with the very 
similar(but still different) codebase?

> Upgrade Spark module to 2.4
> ---------------------------
>
>                 Key: IGNITE-12054
>                 URL: https://issues.apache.org/jira/browse/IGNITE-12054
>             Project: Ignite
>          Issue Type: Task
>          Components: spark
>    Affects Versions: 2.7.5
>            Reporter: Denis Magda
>            Assignee: Nikolay Izhikov
>            Priority: Blocker
>             Fix For: 2.8
>
>
> Users can't use APIs that are already available in Spark 2.4:
> https://stackoverflow.com/questions/57392143/persisting-spark-dataframe-to-ignite
> Let's upgrade Spark from 2.3 to 2.4 until we extract the Spark Integration as 
> a separate module that can support multiple Spark versions.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Reply via email to