[
https://issues.apache.org/jira/browse/IGNITE-3084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16304299#comment-16304299
]
Nikolay Izhikov commented on IGNITE-3084:
-----------------------------------------
{quote}IgniteSQLRelation#calcPartitions - got it, but what will happen if
topology changes? Will partitions be recalculated?{quote}
If Ignite topology changed *after* {{calcPartitions}} executed and *before*
query fully executed then Ignite SQL query for a corresponding Spark partition
will read data from several server Ignite nodes.
Next query will execute {{calcPartitions}} with actual topology info and will
work as expected.
Note, that topology change doesn’t affect results correctness.
Because, we query only 1 table from Ignite and all aggregate, currently,
calculated by Spark.
> Spark Data Frames Support in Apache Ignite
> ------------------------------------------
>
> Key: IGNITE-3084
> URL: https://issues.apache.org/jira/browse/IGNITE-3084
> Project: Ignite
> Issue Type: Task
> Components: spark
> Affects Versions: 1.5.0.final
> Reporter: Vladimir Ozerov
> Assignee: Nikolay Izhikov
> Priority: Critical
> Labels: bigdata, important
> Fix For: 2.4
>
>
> Apache Spark already benefits from integration with Apache Ignite. The latter
> provides shared RDDs, an implementation of Spark RDD, that help Spark to
> share a state between Spark workers and execute SQL queries much faster. The
> next logical step is to enable support for modern Spark Data Frames API in a
> similar way.
> As a contributor, you will be fully in charge of the integration of Spark
> Data Frame API and Apache Ignite.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
