[
https://issues.apache.org/jira/browse/IGNITE-3084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15795597#comment-15795597
]
Valentin Kulichenko commented on IGNITE-3084:
---------------------------------------------
Logical plan (which is actually AST) is built by Spark based on the API calls
you make. It supports both SQL (Spark parses it by itself in this case) and
chain methods like {{filter(..)}}, {{join(..)}}, etc. Logical plan is then
converted to physical plan which defines how the logical plan is actually
executed. So basically we need a strategy that will generate SQL query for
Ignite based on AST provided by Spark.
In addition to this, MemSQL provides an option to execute SQL query as is when
{{SQLContext.sql(..)}} method is called (i.e. it bypasses Spark query
parser/planner). Not sure this is really useful because this implies adding
another method on top of standard API, but it's fairly easy to add, so it make
sense to do the same.
> Investigate how Ignite can support Spark DataFrame
> --------------------------------------------------
>
> Key: IGNITE-3084
> URL: https://issues.apache.org/jira/browse/IGNITE-3084
> Project: Ignite
> Issue Type: Task
> Components: Ignite RDD
> Affects Versions: 1.5.0.final
> Reporter: Vladimir Ozerov
> Assignee: Valentin Kulichenko
> Labels: bigdata
> Fix For: 2.0
>
>
> We see increasing demand on nice DataFrame support for our Spark integration.
> Need to investigate how could we do that.
> Looks like we can investigate how MemSQL do that and take it as a starting
> point.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
