[
https://issues.apache.org/jira/browse/IGNITE-3084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16305057#comment-16305057
]
Nikolay Izhikov commented on IGNITE-3084:
-----------------------------------------
{quote}
I don't agree on OPTION_GRID. In catalog implementation you already have Ignite
instance, so I don't see why would you need to get it by name again. There
should be another solution.
And in general, implementation detail should not drive public API. OPTION_GRID
is very confusing and I still think it should be removed, at least in this
iteration.
{quote}
I made {{OPTION_GRID}} private in IgniteExternalCatalog so it not appear in
public API.
Is it OK?
Please, see some details about spark relation resolving:
1. When spark execute SQL query it resolves catalog relation with
{{FindDataSourceTable}}. Relation resolved with string options provided from
Catalog:
https://github.com/apache/spark/blob/v2.2.0/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala#L230
2. DataSource constructs relation throw
RelationProvider(IgniteRelationProvider) with these string options:
https://github.com/apache/spark/blob/v2.2.0/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala#L306
So we have to pass some string ID from a catalog to IgniteRelationProvider to
get existing Ignite instance.
As far as I can understand - Ignite.name is well for this purpose.
What have I missed?
Another option, that I can see - make some internal HashMap[String, Ignite] and
use `UUID.randomUUID()` as Ignite identifier.
What do you think?
> Spark Data Frames Support in Apache Ignite
> ------------------------------------------
>
> Key: IGNITE-3084
> URL: https://issues.apache.org/jira/browse/IGNITE-3084
> Project: Ignite
> Issue Type: Task
> Components: spark
> Affects Versions: 1.5.0.final
> Reporter: Vladimir Ozerov
> Assignee: Nikolay Izhikov
> Priority: Critical
> Labels: bigdata, important
> Fix For: 2.4
>
>
> Apache Spark already benefits from integration with Apache Ignite. The latter
> provides shared RDDs, an implementation of Spark RDD, that help Spark to
> share a state between Spark workers and execute SQL queries much faster. The
> next logical step is to enable support for modern Spark Data Frames API in a
> similar way.
> As a contributor, you will be fully in charge of the integration of Spark
> Data Frame API and Apache Ignite.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
