[
https://issues.apache.org/jira/browse/IGNITE-12054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16937715#comment-16937715
]
Aleksey Zinoviev edited comment on IGNITE-12054 at 9/25/19 1:37 PM:
--------------------------------------------------------------------
I've started the R&D about 2.4 version, fixed a few minor changes and posted
here a few thoughts about current problems here, in Jira comments.
An ExternalCatalog was refactored here
[https://github.com/apache/spark/commit/f38ea00e83099a5ae8d3afdec2e896e43c2db612]
and all listener properties were inherited in ExternalCatalogWithListener.
Nobody yet inherited from this class on Github, the known implementations are
HiveExternalCatalog and MemoryExternalCatalog (both of them doesn't support
listeners and events)
Also, people in Spark ML couldn't solve the same problem
[http://mail-archives.apache.org/mod_mbox/spark-issues/201812.mbox/%3cjira.13144856.1520975543000.147283.1544598241...@atlassian.jira%3E]
Also this question is considered in paper
[https://www.waitingforcode.com/apache-spark-sql/writing-custom-external-catalog-listeners-apache-spark-sql/read]
was (Author: zaleslaw):
I've started the R&D about 2.4 version, fixed a few minor changes and posted
here a few thoughts about current problems here, in Jira comments.
An ExternalCatalog was refactored here
[https://github.com/apache/spark/commit/f38ea00e83099a5ae8d3afdec2e896e43c2db612]
and all listener properties were inherited in ExternalCatalogWithListener.
Nobody yet inherited from this class on Github, the known implementations are
HiveExternalCatalog and MemoryExternalCatalog (both of them doesn't support
listeners and events)
Also, people in Spark ML couldn't solve the same problem
[http://mail-archives.apache.org/mod_mbox/spark-issues/201812.mbox/%3cjira.13144856.1520975543000.147283.1544598241...@atlassian.jira%3E]
> Upgrade Spark module to 2.4
> ---------------------------
>
> Key: IGNITE-12054
> URL: https://issues.apache.org/jira/browse/IGNITE-12054
> Project: Ignite
> Issue Type: Task
> Components: spark
> Affects Versions: 2.7.5
> Reporter: Denis Magda
> Assignee: Aleksey Zinoviev
> Priority: Blocker
> Fix For: 2.8
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Users can't use APIs that are already available in Spark 2.4:
> https://stackoverflow.com/questions/57392143/persisting-spark-dataframe-to-ignite
> Let's upgrade Spark from 2.3 to 2.4 until we extract the Spark Integration as
> a separate module that can support multiple Spark versions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
