[jira] [Commented] (IGNITE-7077) Spark Data Frame Support. Strategy to convert complete query to Ignite SQL
[ https://issues.apache.org/jira/browse/IGNITE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16601259#comment-16601259 ] Valentin Kulichenko commented on IGNITE-7077: - [~lzzukowski], you can join different dataframes with each other, so you can have an Ignite dataframe joined with a non-Ignite dataframe. You will not take advantage of all performance optimization Ignite integration provides, why it should work. P.S. Going forward, please ask such questions on the user list. > Spark Data Frame Support. Strategy to convert complete query to Ignite SQL > -- > > Key: IGNITE-7077 > URL: https://issues.apache.org/jira/browse/IGNITE-7077 > Project: Ignite > Issue Type: New Feature > Components: spark >Affects Versions: 2.3 >Reporter: Nikolay Izhikov >Assignee: Nikolay Izhikov >Priority: Major > Labels: bigdata > Fix For: 2.5 > > > Basic support of Spark Data Frame for Ignite implemented in IGNITE-3084. > We need to implement custom spark strategy that can convert whole Spark SQL > query to Ignite SQL Query if query consists of only Ignite tables. > The strategy does nothing if spark query includes not only Ignite tables. > Memsql implementation can be taken as an example - > https://github.com/memsql/memsql-spark-connector -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7077) Spark Data Frame Support. Strategy to convert complete query to Ignite SQL
[ https://issues.apache.org/jira/browse/IGNITE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16601189#comment-16601189 ] Łukasz Żukowski commented on IGNITE-7077: - Hi I have question about spark joins in Ignite. It is possible to do something similiar to joinWithCassandraTable described below [https://github.com/datastax/spark-cassandra-connector/blob/master/doc/2_loading.md] In general I want create query to Ignite Cluster for eache record in Dataframe/RDD and put result as joined collection. It is very usefull where we have large dataset but join is only for small subset of data eg. in streaming. Or maybe in Ignite Spark integration this can be done in other way. > Spark Data Frame Support. Strategy to convert complete query to Ignite SQL > -- > > Key: IGNITE-7077 > URL: https://issues.apache.org/jira/browse/IGNITE-7077 > Project: Ignite > Issue Type: New Feature > Components: spark >Affects Versions: 2.3 >Reporter: Nikolay Izhikov >Assignee: Nikolay Izhikov >Priority: Major > Labels: bigdata > Fix For: 2.5 > > > Basic support of Spark Data Frame for Ignite implemented in IGNITE-3084. > We need to implement custom spark strategy that can convert whole Spark SQL > query to Ignite SQL Query if query consists of only Ignite tables. > The strategy does nothing if spark query includes not only Ignite tables. > Memsql implementation can be taken as an example - > https://github.com/memsql/memsql-spark-connector -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7077) Spark Data Frame Support. Strategy to convert complete query to Ignite SQL
[ https://issues.apache.org/jira/browse/IGNITE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16448090#comment-16448090 ] Dmitriy Pavlov commented on IGNITE-7077: It seems ok for me > Spark Data Frame Support. Strategy to convert complete query to Ignite SQL > -- > > Key: IGNITE-7077 > URL: https://issues.apache.org/jira/browse/IGNITE-7077 > Project: Ignite > Issue Type: New Feature > Components: spark >Affects Versions: 2.3 >Reporter: Nikolay Izhikov >Assignee: Nikolay Izhikov >Priority: Major > Labels: bigdata > Fix For: 2.5 > > > Basic support of Spark Data Frame for Ignite implemented in IGNITE-3084. > We need to implement custom spark strategy that can convert whole Spark SQL > query to Ignite SQL Query if query consists of only Ignite tables. > The strategy does nothing if spark query includes not only Ignite tables. > Memsql implementation can be taken as an example - > https://github.com/memsql/memsql-spark-connector -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7077) Spark Data Frame Support. Strategy to convert complete query to Ignite SQL
[ https://issues.apache.org/jira/browse/IGNITE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16448057#comment-16448057 ] Nikolay Izhikov commented on IGNITE-7077: - Two commits added to 2.5 branch https://github.com/apache/ignite/commit/3c3a24e84a9a9f07d0833b29524de244930634c3 https://github.com/apache/ignite/commit/d7962f83f14292a571016b79af594ceefe632d83 Please, check. I done it for the first time. > Spark Data Frame Support. Strategy to convert complete query to Ignite SQL > -- > > Key: IGNITE-7077 > URL: https://issues.apache.org/jira/browse/IGNITE-7077 > Project: Ignite > Issue Type: New Feature > Components: spark >Affects Versions: 2.3 >Reporter: Nikolay Izhikov >Assignee: Nikolay Izhikov >Priority: Major > Labels: bigdata > Fix For: 2.5 > > > Basic support of Spark Data Frame for Ignite implemented in IGNITE-3084. > We need to implement custom spark strategy that can convert whole Spark SQL > query to Ignite SQL Query if query consists of only Ignite tables. > The strategy does nothing if spark query includes not only Ignite tables. > Memsql implementation can be taken as an example - > https://github.com/memsql/memsql-spark-connector -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7077) Spark Data Frame Support. Strategy to convert complete query to Ignite SQL
[ https://issues.apache.org/jira/browse/IGNITE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447968#comment-16447968 ] Dmitriy Pavlov commented on IGNITE-7077: I'm not aware of instructions, but usually I find commit in master log and perform cherry-picking it to release branch > Spark Data Frame Support. Strategy to convert complete query to Ignite SQL > -- > > Key: IGNITE-7077 > URL: https://issues.apache.org/jira/browse/IGNITE-7077 > Project: Ignite > Issue Type: New Feature > Components: spark >Affects Versions: 2.3 >Reporter: Nikolay Izhikov >Assignee: Nikolay Izhikov >Priority: Major > Labels: bigdata > Fix For: 2.5 > > > Basic support of Spark Data Frame for Ignite implemented in IGNITE-3084. > We need to implement custom spark strategy that can convert whole Spark SQL > query to Ignite SQL Query if query consists of only Ignite tables. > The strategy does nothing if spark query includes not only Ignite tables. > Memsql implementation can be taken as an example - > https://github.com/memsql/memsql-spark-connector -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7077) Spark Data Frame Support. Strategy to convert complete query to Ignite SQL
[ https://issues.apache.org/jira/browse/IGNITE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16446680#comment-16446680 ] Nikolay Izhikov commented on IGNITE-7077: - [~vkulichenko] > Nikolay, can you please do this? Sure. I didn't do that earlier. Is there any instruction how to merge commits into release branch? > Spark Data Frame Support. Strategy to convert complete query to Ignite SQL > -- > > Key: IGNITE-7077 > URL: https://issues.apache.org/jira/browse/IGNITE-7077 > Project: Ignite > Issue Type: New Feature > Components: spark >Affects Versions: 2.3 >Reporter: Nikolay Izhikov >Assignee: Nikolay Izhikov >Priority: Major > Labels: bigdata > Fix For: 2.5 > > > Basic support of Spark Data Frame for Ignite implemented in IGNITE-3084. > We need to implement custom spark strategy that can convert whole Spark SQL > query to Ignite SQL Query if query consists of only Ignite tables. > The strategy does nothing if spark query includes not only Ignite tables. > Memsql implementation can be taken as an example - > https://github.com/memsql/memsql-spark-connector -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7077) Spark Data Frame Support. Strategy to convert complete query to Ignite SQL
[
https://issues.apache.org/jira/browse/IGNITE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16446178#comment-16446178
]
Valentin Kulichenko commented on IGNITE-7077:
-
[~dpavlov], [~NIzhikov], [~agura], guys, this was planned for 2.5 according to
dev list discussions. Let's merge it to {{ignite-2.5}}. Nikolay, can you please
do this?
> Spark Data Frame Support. Strategy to convert complete query to Ignite SQL
> --
>
> Key: IGNITE-7077
> URL: https://issues.apache.org/jira/browse/IGNITE-7077
> Project: Ignite
> Issue Type: New Feature
> Components: spark
>Affects Versions: 2.3
>Reporter: Nikolay Izhikov
>Assignee: Nikolay Izhikov
>Priority: Major
> Labels: bigdata
> Fix For: 2.5
>
>
> Basic support of Spark Data Frame for Ignite implemented in IGNITE-3084.
> We need to implement custom spark strategy that can convert whole Spark SQL
> query to Ignite SQL Query if query consists of only Ignite tables.
> The strategy does nothing if spark query includes not only Ignite tables.
> Memsql implementation can be taken as an example -
> https://github.com/memsql/memsql-spark-connector
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (IGNITE-7077) Spark Data Frame Support. Strategy to convert complete query to Ignite SQL
[ https://issues.apache.org/jira/browse/IGNITE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16445982#comment-16445982 ] Nikolay Izhikov commented on IGNITE-7077: - [~agoncharuk] Thank you! > Spark Data Frame Support. Strategy to convert complete query to Ignite SQL > -- > > Key: IGNITE-7077 > URL: https://issues.apache.org/jira/browse/IGNITE-7077 > Project: Ignite > Issue Type: New Feature > Components: spark >Affects Versions: 2.3 >Reporter: Nikolay Izhikov >Assignee: Nikolay Izhikov >Priority: Major > Labels: bigdata > Fix For: 2.5 > > > Basic support of Spark Data Frame for Ignite implemented in IGNITE-3084. > We need to implement custom spark strategy that can convert whole Spark SQL > query to Ignite SQL Query if query consists of only Ignite tables. > The strategy does nothing if spark query includes not only Ignite tables. > Memsql implementation can be taken as an example - > https://github.com/memsql/memsql-spark-connector -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7077) Spark Data Frame Support. Strategy to convert complete query to Ignite SQL
[ https://issues.apache.org/jira/browse/IGNITE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16445965#comment-16445965 ] Dmitriy Pavlov commented on IGNITE-7077: [~NIzhikov] please also fix licenses before cherry-picking [https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_LicensesHeaders=%3Cdefault%3E=buildTypeStatusDiv] > Spark Data Frame Support. Strategy to convert complete query to Ignite SQL > -- > > Key: IGNITE-7077 > URL: https://issues.apache.org/jira/browse/IGNITE-7077 > Project: Ignite > Issue Type: New Feature > Components: spark >Affects Versions: 2.3 >Reporter: Nikolay Izhikov >Assignee: Nikolay Izhikov >Priority: Major > Labels: bigdata > Fix For: 2.5 > > > Basic support of Spark Data Frame for Ignite implemented in IGNITE-3084. > We need to implement custom spark strategy that can convert whole Spark SQL > query to Ignite SQL Query if query consists of only Ignite tables. > The strategy does nothing if spark query includes not only Ignite tables. > Memsql implementation can be taken as an example - > https://github.com/memsql/memsql-spark-connector -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7077) Spark Data Frame Support. Strategy to convert complete query to Ignite SQL
[ https://issues.apache.org/jira/browse/IGNITE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16445977#comment-16445977 ] Alexey Goncharuk commented on IGNITE-7077: -- Fixed licenses, please do not forget to cherry-pick commit fa841242a3ce7d5179e33663a724f3c3c07ef673 to ignite-2.5 if this ticket should be included to ignite-2.5 > Spark Data Frame Support. Strategy to convert complete query to Ignite SQL > -- > > Key: IGNITE-7077 > URL: https://issues.apache.org/jira/browse/IGNITE-7077 > Project: Ignite > Issue Type: New Feature > Components: spark >Affects Versions: 2.3 >Reporter: Nikolay Izhikov >Assignee: Nikolay Izhikov >Priority: Major > Labels: bigdata > Fix For: 2.5 > > > Basic support of Spark Data Frame for Ignite implemented in IGNITE-3084. > We need to implement custom spark strategy that can convert whole Spark SQL > query to Ignite SQL Query if query consists of only Ignite tables. > The strategy does nothing if spark query includes not only Ignite tables. > Memsql implementation can be taken as an example - > https://github.com/memsql/memsql-spark-connector -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7077) Spark Data Frame Support. Strategy to convert complete query to Ignite SQL
[ https://issues.apache.org/jira/browse/IGNITE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16445740#comment-16445740 ] Dmitriy Pavlov commented on IGNITE-7077: [~vkulichenko], thank you for review. > Spark Data Frame Support. Strategy to convert complete query to Ignite SQL > -- > > Key: IGNITE-7077 > URL: https://issues.apache.org/jira/browse/IGNITE-7077 > Project: Ignite > Issue Type: New Feature > Components: spark >Affects Versions: 2.3 >Reporter: Nikolay Izhikov >Assignee: Nikolay Izhikov >Priority: Major > Labels: bigdata > Fix For: 2.5 > > > Basic support of Spark Data Frame for Ignite implemented in IGNITE-3084. > We need to implement custom spark strategy that can convert whole Spark SQL > query to Ignite SQL Query if query consists of only Ignite tables. > The strategy does nothing if spark query includes not only Ignite tables. > Memsql implementation can be taken as an example - > https://github.com/memsql/memsql-spark-connector -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7077) Spark Data Frame Support. Strategy to convert complete query to Ignite SQL
[ https://issues.apache.org/jira/browse/IGNITE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16445738#comment-16445738 ] Dmitriy Pavlov commented on IGNITE-7077: [~NIzhikov] please set version 2.6 in JIRA, as it seems that change is not included into branch ignite-2.5. Alternatively cherry-pick commit to ignite-2.5 if [~agura] agree. > Spark Data Frame Support. Strategy to convert complete query to Ignite SQL > -- > > Key: IGNITE-7077 > URL: https://issues.apache.org/jira/browse/IGNITE-7077 > Project: Ignite > Issue Type: New Feature > Components: spark >Affects Versions: 2.3 >Reporter: Nikolay Izhikov >Assignee: Nikolay Izhikov >Priority: Major > Labels: bigdata > Fix For: 2.5 > > > Basic support of Spark Data Frame for Ignite implemented in IGNITE-3084. > We need to implement custom spark strategy that can convert whole Spark SQL > query to Ignite SQL Query if query consists of only Ignite tables. > The strategy does nothing if spark query includes not only Ignite tables. > Memsql implementation can be taken as an example - > https://github.com/memsql/memsql-spark-connector -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7077) Spark Data Frame Support. Strategy to convert complete query to Ignite SQL
[ https://issues.apache.org/jira/browse/IGNITE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16445508#comment-16445508 ] ASF GitHub Bot commented on IGNITE-7077: Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/3397 > Spark Data Frame Support. Strategy to convert complete query to Ignite SQL > -- > > Key: IGNITE-7077 > URL: https://issues.apache.org/jira/browse/IGNITE-7077 > Project: Ignite > Issue Type: New Feature > Components: spark >Affects Versions: 2.3 >Reporter: Nikolay Izhikov >Assignee: Nikolay Izhikov >Priority: Major > Labels: bigdata > Fix For: 2.5 > > > Basic support of Spark Data Frame for Ignite implemented in IGNITE-3084. > We need to implement custom spark strategy that can convert whole Spark SQL > query to Ignite SQL Query if query consists of only Ignite tables. > The strategy does nothing if spark query includes not only Ignite tables. > Memsql implementation can be taken as an example - > https://github.com/memsql/memsql-spark-connector -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7077) Spark Data Frame Support. Strategy to convert complete query to Ignite SQL
[ https://issues.apache.org/jira/browse/IGNITE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16445511#comment-16445511 ] Nikolay Izhikov commented on IGNITE-7077: - merged to master [~vkulichenko] Thank you, very much! > Spark Data Frame Support. Strategy to convert complete query to Ignite SQL > -- > > Key: IGNITE-7077 > URL: https://issues.apache.org/jira/browse/IGNITE-7077 > Project: Ignite > Issue Type: New Feature > Components: spark >Affects Versions: 2.3 >Reporter: Nikolay Izhikov >Assignee: Nikolay Izhikov >Priority: Major > Labels: bigdata > Fix For: 2.5 > > > Basic support of Spark Data Frame for Ignite implemented in IGNITE-3084. > We need to implement custom spark strategy that can convert whole Spark SQL > query to Ignite SQL Query if query consists of only Ignite tables. > The strategy does nothing if spark query includes not only Ignite tables. > Memsql implementation can be taken as an example - > https://github.com/memsql/memsql-spark-connector -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7077) Spark Data Frame Support. Strategy to convert complete query to Ignite SQL
[ https://issues.apache.org/jira/browse/IGNITE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16444600#comment-16444600 ] Valentin Kulichenko commented on IGNITE-7077: - [~NIzhikov], OK, this naming makes sense to me then. I believe it's good to merge. Thanks! > Spark Data Frame Support. Strategy to convert complete query to Ignite SQL > -- > > Key: IGNITE-7077 > URL: https://issues.apache.org/jira/browse/IGNITE-7077 > Project: Ignite > Issue Type: New Feature > Components: spark >Affects Versions: 2.3 >Reporter: Nikolay Izhikov >Assignee: Nikolay Izhikov >Priority: Major > Labels: bigdata > Fix For: 2.5 > > > Basic support of Spark Data Frame for Ignite implemented in IGNITE-3084. > We need to implement custom spark strategy that can convert whole Spark SQL > query to Ignite SQL Query if query consists of only Ignite tables. > The strategy does nothing if spark query includes not only Ignite tables. > Memsql implementation can be taken as an example - > https://github.com/memsql/memsql-spark-connector -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7077) Spark Data Frame Support. Strategy to convert complete query to Ignite SQL
[ https://issues.apache.org/jira/browse/IGNITE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440599#comment-16440599 ] Nikolay Izhikov commented on IGNITE-7077: - TC https://ci.ignite.apache.org/viewQueued.html?itemId=1215749=queuedBuildOverviewTab https://ci.ignite.apache.org/viewQueued.html?itemId=1215750=queuedBuildOverviewTab > Spark Data Frame Support. Strategy to convert complete query to Ignite SQL > -- > > Key: IGNITE-7077 > URL: https://issues.apache.org/jira/browse/IGNITE-7077 > Project: Ignite > Issue Type: New Feature > Components: spark >Affects Versions: 2.3 >Reporter: Nikolay Izhikov >Assignee: Nikolay Izhikov >Priority: Major > Labels: bigdata > Fix For: 2.5 > > > Basic support of Spark Data Frame for Ignite implemented in IGNITE-3084. > We need to implement custom spark strategy that can convert whole Spark SQL > query to Ignite SQL Query if query consists of only Ignite tables. > The strategy does nothing if spark query includes not only Ignite tables. > Memsql implementation can be taken as an example - > https://github.com/memsql/memsql-spark-connector -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7077) Spark Data Frame Support. Strategy to convert complete query to Ignite SQL
[
https://issues.apache.org/jira/browse/IGNITE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440509#comment-16440509
]
Nikolay Izhikov commented on IGNITE-7077:
-
[~vkulichenko]
> I noticed that the new setting name has ignite. prefix. I generally like
> this, but this is not consistent with others.
All other options are used for an *Ignite* Data Frame configuration.
I think we don't need an additional prefix to be passed straight to the
{{IgniteRelationProvider}}.
{{OPTION_DISABLE_SPARK_SQL_OPTIMIZATION}} - option to be used when *Spark
session* is configured.
So I made this option consistent with other spark session options.
{{"spark.sql.inMemoryColumnarStorage.compressed"}}, etc [1]
> Is it possible to change all others in the same way? If yes, let's do this.
> Otherwise, let's remove the prefix.
I think we shouldn't change existing constant values.
It's a part of public API and we released it.
Personally, I like current naming.
Anyway, I can remove prefix in {{OPTION_DISABLE_SPARK_SQL_OPTIMIZATION}}, if
you want.
Do you like {{"disableIgniteSparkSQLOptimization"}}?
> Out of curiosity, what is the purpose of the code below? Shouldn't we just do
> nothing if optimization is disabled?
Thank you. It's a bug in my PR and I fixed it.
Spark Session configuration can be changed at runtime.
So we have to check if Ignite optimization enabled each time query occurs.
And if it disabled we have to remove {{IgniteOptimization}} from extra
optimizations.
Correct code is
{{code:scala}}
if (optimizationDisabled) {
experimentalMethods.extraOptimizations =
experimentalMethods.extraOptimizations.filter(_ !=
IgniteOptimization)
} else {
val optimizationExists =
experimentalMethods.extraOptimizations.contains(IgniteOptimization)
if (!optimizationExists)
experimentalMethods.extraOptimizations =
experimentalMethods.extraOptimizations :+ IgniteOptimization
}
{{code}}
Please, see updated PR
[1]
[https://github.com/apache/spark/blob/v2.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L80]
> Spark Data Frame Support. Strategy to convert complete query to Ignite SQL
> --
>
> Key: IGNITE-7077
> URL: https://issues.apache.org/jira/browse/IGNITE-7077
> Project: Ignite
> Issue Type: New Feature
> Components: spark
>Affects Versions: 2.3
>Reporter: Nikolay Izhikov
>Assignee: Nikolay Izhikov
>Priority: Major
> Labels: bigdata
> Fix For: 2.5
>
>
> Basic support of Spark Data Frame for Ignite implemented in IGNITE-3084.
> We need to implement custom spark strategy that can convert whole Spark SQL
> query to Ignite SQL Query if query consists of only Ignite tables.
> The strategy does nothing if spark query includes not only Ignite tables.
> Memsql implementation can be taken as an example -
> https://github.com/memsql/memsql-spark-connector
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (IGNITE-7077) Spark Data Frame Support. Strategy to convert complete query to Ignite SQL
[
https://issues.apache.org/jira/browse/IGNITE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440351#comment-16440351
]
Valentin Kulichenko commented on IGNITE-7077:
-
[~NIzhikov], several minor comments.
# I noticed that the new setting name has {{ignite.}} prefix. I generally like
this, but this is not consistent with others. Is it possible to change all
others in the same way? If yes, let's do this. Otherwise, let's remove the
prefix.
# Is {{IgniteOptimization}} a public class? If yes, is this required? If not
required, is it possible to avoid that?
# Out of curiosity, what is the purpose of the code below? Shouldn't we just do
nothing if optimization is disabled?
{code}
if (optimizationDisabled)
experimentalMethods.extraOptimizations.filter(_ != IgniteOptimization)
{code}
> Spark Data Frame Support. Strategy to convert complete query to Ignite SQL
> --
>
> Key: IGNITE-7077
> URL: https://issues.apache.org/jira/browse/IGNITE-7077
> Project: Ignite
> Issue Type: New Feature
> Components: spark
>Affects Versions: 2.3
>Reporter: Nikolay Izhikov
>Assignee: Nikolay Izhikov
>Priority: Major
> Labels: bigdata
> Fix For: 2.5
>
>
> Basic support of Spark Data Frame for Ignite implemented in IGNITE-3084.
> We need to implement custom spark strategy that can convert whole Spark SQL
> query to Ignite SQL Query if query consists of only Ignite tables.
> The strategy does nothing if spark query includes not only Ignite tables.
> Memsql implementation can be taken as an example -
> https://github.com/memsql/memsql-spark-connector
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (IGNITE-7077) Spark Data Frame Support. Strategy to convert complete query to Ignite SQL
[ https://issues.apache.org/jira/browse/IGNITE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16434153#comment-16434153 ] Nikolay Izhikov commented on IGNITE-7077: - Run all results - https://ci.ignite.apache.org/viewLog.html?buildId=1191704=queuedBuildOverviewTab > Spark Data Frame Support. Strategy to convert complete query to Ignite SQL > -- > > Key: IGNITE-7077 > URL: https://issues.apache.org/jira/browse/IGNITE-7077 > Project: Ignite > Issue Type: New Feature > Components: spark >Affects Versions: 2.3 >Reporter: Nikolay Izhikov >Assignee: Nikolay Izhikov >Priority: Major > Labels: bigdata > Fix For: 2.5 > > > Basic support of Spark Data Frame for Ignite implemented in IGNITE-3084. > We need to implement custom spark strategy that can convert whole Spark SQL > query to Ignite SQL Query if query consists of only Ignite tables. > The strategy does nothing if spark query includes not only Ignite tables. > Memsql implementation can be taken as an example - > https://github.com/memsql/memsql-spark-connector -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7077) Spark Data Frame Support. Strategy to convert complete query to Ignite SQL
[ https://issues.apache.org/jira/browse/IGNITE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16433620#comment-16433620 ] Nikolay Izhikov commented on IGNITE-7077: - Hello, [~vkulichenko]. Thank you for review. I've fixed PR according to your comments. Please, take a look. > Spark Data Frame Support. Strategy to convert complete query to Ignite SQL > -- > > Key: IGNITE-7077 > URL: https://issues.apache.org/jira/browse/IGNITE-7077 > Project: Ignite > Issue Type: New Feature > Components: spark >Affects Versions: 2.3 >Reporter: Nikolay Izhikov >Assignee: Nikolay Izhikov >Priority: Major > Labels: bigdata > Fix For: 2.5 > > > Basic support of Spark Data Frame for Ignite implemented in IGNITE-3084. > We need to implement custom spark strategy that can convert whole Spark SQL > query to Ignite SQL Query if query consists of only Ignite tables. > The strategy does nothing if spark query includes not only Ignite tables. > Memsql implementation can be taken as an example - > https://github.com/memsql/memsql-spark-connector -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7077) Spark Data Frame Support. Strategy to convert complete query to Ignite SQL
[
https://issues.apache.org/jira/browse/IGNITE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16433216#comment-16433216
]
Valentin Kulichenko commented on IGNITE-7077:
-
[~NIzhikov], overall looks good, but I have couple comments:
* Let's try to make sure optimization is always enabled, not only when
{{IgniteSparkSession}} is used (in other words, code in
{{IgniteDataFrameExample}} should utilize the optimization).
* I think we should have a configuration parameter that would disable the
optimization. By default it should be enabled of course.
> Spark Data Frame Support. Strategy to convert complete query to Ignite SQL
> --
>
> Key: IGNITE-7077
> URL: https://issues.apache.org/jira/browse/IGNITE-7077
> Project: Ignite
> Issue Type: New Feature
> Components: spark
>Affects Versions: 2.3
>Reporter: Nikolay Izhikov
>Assignee: Nikolay Izhikov
>Priority: Major
> Labels: bigdata
> Fix For: 2.5
>
>
> Basic support of Spark Data Frame for Ignite implemented in IGNITE-3084.
> We need to implement custom spark strategy that can convert whole Spark SQL
> query to Ignite SQL Query if query consists of only Ignite tables.
> The strategy does nothing if spark query includes not only Ignite tables.
> Memsql implementation can be taken as an example -
> https://github.com/memsql/memsql-spark-connector
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (IGNITE-7077) Spark Data Frame Support. Strategy to convert complete query to Ignite SQL
[
https://issues.apache.org/jira/browse/IGNITE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427949#comment-16427949
]
Nikolay Izhikov commented on IGNITE-7077:
-
[~vkulichenko]
First, thank you for paying attention to my task!
> I thought this task implied implementation of Strategy to convert Spark's
> logical plan to physical plan that would be executed directly on Ignite as a
> SQL query.
1. LogicalPlan is the representation of SQL query.
2. Leave of LogicalPlan is wrapper for {{IgniteSQLRealtion}}.
3. As far as I can see we don't need to provide some custom {{SparkStrategy}}.
At least for a query optimization described in this issue.
Because {{SparkStrategy}} contains Spark 'physical' operations: how to select
and process data from the underlying data sources optimally.
This differs from {{LogicalPlan}} - tree representation of SQL query.
> Here I see the implementation of Optimization. Can you please clarify why is
> that and what is the difference? How the current implementation work?
My implementation does the following:
1. Transform part(or the whole) LogicalPlan from the bottom to the up and
pushes all possible SQL operator to the Ignite accumulator.
If operator pushed to the accumulator then tree node are removed from the plan,
because we execute it internally in Iginte.
2. If on some level unsupported operator founded(some UDF or similar) then this
and upper layers of LogicalPlan remains unchanged.
3. Create SQL query from accumulator. It will be executed directly in Ignite.
4. Replace accumulator with a `IgniteSQLAccumulatorRelation` which is relation
containing resulting SQL query.
After step 4 we have a LogicalPlan that will execute all supported SQL operator
directly in Ignite.
> Also I think we should add some examples demonstrating the new functionality.
If it works correctly then we don't need any new examples, because
implementation details are hidden from the user.
Here what changed:
This how spark plan looks like *before* Ignite optimization:
{noformat}
== Analyzed Logical Plan ==
id: bigint, name: string
Sort [id#180L ASC NULLS FIRST], true
+- Project [id#180L, name#179]
+- Filter (id#180L > cast(1 as bigint))
+- SubqueryAlias city
+- Relation[NAME#179,ID#180L] IgniteSQLRelation[table=CITY]
{noformat}
After optimization, we got following plan. Please note, that query inside
IgniteSQLAccumulatorRelation identical to the query representation from the
previous plan.
{noformat}
== Optimized Logical Plan ==
Relation[ID#180L,NAME#179] IgniteSQLAccumulatorRelation(columns=[ID, NAME],
qry=SELECT ID, NAME FROM CITY WHERE id IS NOT NULL AND id > 1 ORDER BY id)
{noformat}
Please, see debug output of IgniteOptimization*Spec. {{IgniteOptimizationSpec}}
or {{IgniteOptimizationStringFuncSpec}}, for example.
You can see additional examples of LogicalPlan transformation there.
> Spark Data Frame Support. Strategy to convert complete query to Ignite SQL
> --
>
> Key: IGNITE-7077
> URL: https://issues.apache.org/jira/browse/IGNITE-7077
> Project: Ignite
> Issue Type: New Feature
> Components: spark
>Affects Versions: 2.3
>Reporter: Nikolay Izhikov
>Assignee: Nikolay Izhikov
>Priority: Major
> Labels: bigdata
> Fix For: 2.5
>
>
> Basic support of Spark Data Frame for Ignite implemented in IGNITE-3084.
> We need to implement custom spark strategy that can convert whole Spark SQL
> query to Ignite SQL Query if query consists of only Ignite tables.
> The strategy does nothing if spark query includes not only Ignite tables.
> Memsql implementation can be taken as an example -
> https://github.com/memsql/memsql-spark-connector
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (IGNITE-7077) Spark Data Frame Support. Strategy to convert complete query to Ignite SQL
[
https://issues.apache.org/jira/browse/IGNITE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427797#comment-16427797
]
Valentin Kulichenko commented on IGNITE-7077:
-
[~NIzhikov], I'm a bit confused. I thought this task implied implementation of
{{Strategy}} to convert Spark's logical plan to physical plan that would be
executed directly on Ignite as a SQL query. Here I see the implementation of
{{Optimization}}. Can you please clarify why is that and what is the
difference? How the current implementation work?
Also I think we should add some examples demonstrating the new functionality.
> Spark Data Frame Support. Strategy to convert complete query to Ignite SQL
> --
>
> Key: IGNITE-7077
> URL: https://issues.apache.org/jira/browse/IGNITE-7077
> Project: Ignite
> Issue Type: New Feature
> Components: spark
>Affects Versions: 2.3
>Reporter: Nikolay Izhikov
>Assignee: Nikolay Izhikov
>Priority: Major
> Labels: bigdata
> Fix For: 2.5
>
>
> Basic support of Spark Data Frame for Ignite implemented in IGNITE-3084.
> We need to implement custom spark strategy that can convert whole Spark SQL
> query to Ignite SQL Query if query consists of only Ignite tables.
> The strategy does nothing if spark query includes not only Ignite tables.
> Memsql implementation can be taken as an example -
> https://github.com/memsql/memsql-spark-connector
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (IGNITE-7077) Spark Data Frame Support. Strategy to convert complete query to Ignite SQL
[ https://issues.apache.org/jira/browse/IGNITE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401453#comment-16401453 ] Nikolay Izhikov commented on IGNITE-7077: - Hello, [~vkulichenko]! Kind remainder. Can you step in and look at my changes? > Spark Data Frame Support. Strategy to convert complete query to Ignite SQL > -- > > Key: IGNITE-7077 > URL: https://issues.apache.org/jira/browse/IGNITE-7077 > Project: Ignite > Issue Type: New Feature > Components: spark >Affects Versions: 2.3 >Reporter: Nikolay Izhikov >Assignee: Nikolay Izhikov >Priority: Major > Labels: bigdata > Fix For: 2.5 > > > Basic support of Spark Data Frame for Ignite implemented in IGNITE-3084. > We need to implement custom spark strategy that can convert whole Spark SQL > query to Ignite SQL Query if query consists of only Ignite tables. > The strategy does nothing if spark query includes not only Ignite tables. > Memsql implementation can be taken as an example - > https://github.com/memsql/memsql-spark-connector -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7077) Spark Data Frame Support. Strategy to convert complete query to Ignite SQL
[ https://issues.apache.org/jira/browse/IGNITE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386650#comment-16386650 ] Valentin Kulichenko commented on IGNITE-7077: - [~NIzhikov], not yet unfortunately. Will do my best to review this week. Thanks for following up! > Spark Data Frame Support. Strategy to convert complete query to Ignite SQL > -- > > Key: IGNITE-7077 > URL: https://issues.apache.org/jira/browse/IGNITE-7077 > Project: Ignite > Issue Type: New Feature > Components: spark >Affects Versions: 2.3 >Reporter: Nikolay Izhikov >Assignee: Nikolay Izhikov >Priority: Major > Labels: bigdata > Fix For: 2.5 > > > Basic support of Spark Data Frame for Ignite implemented in IGNITE-3084. > We need to implement custom spark strategy that can convert whole Spark SQL > query to Ignite SQL Query if query consists of only Ignite tables. > The strategy does nothing if spark query includes not only Ignite tables. > Memsql implementation can be taken as an example - > https://github.com/memsql/memsql-spark-connector -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7077) Spark Data Frame Support. Strategy to convert complete query to Ignite SQL
[ https://issues.apache.org/jira/browse/IGNITE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16385934#comment-16385934 ] Nikolay Izhikov commented on IGNITE-7077: - Hello [~vkulichenko]. Did you have a chance to look at my PR? > Spark Data Frame Support. Strategy to convert complete query to Ignite SQL > -- > > Key: IGNITE-7077 > URL: https://issues.apache.org/jira/browse/IGNITE-7077 > Project: Ignite > Issue Type: New Feature > Components: spark >Affects Versions: 2.3 >Reporter: Nikolay Izhikov >Assignee: Nikolay Izhikov >Priority: Major > Labels: bigdata > Fix For: 2.5 > > > Basic support of Spark Data Frame for Ignite implemented in IGNITE-3084. > We need to implement custom spark strategy that can convert whole Spark SQL > query to Ignite SQL Query if query consists of only Ignite tables. > The strategy does nothing if spark query includes not only Ignite tables. > Memsql implementation can be taken as an example - > https://github.com/memsql/memsql-spark-connector -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7077) Spark Data Frame Support. Strategy to convert complete query to Ignite SQL
[ https://issues.apache.org/jira/browse/IGNITE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331931#comment-16331931 ] Nikolay Izhikov commented on IGNITE-7077: - [~vozerov], Yes. I share results of my work today. But I still have some issues with implementation and, of course, need the review from commiters. > Spark Data Frame Support. Strategy to convert complete query to Ignite SQL > -- > > Key: IGNITE-7077 > URL: https://issues.apache.org/jira/browse/IGNITE-7077 > Project: Ignite > Issue Type: New Feature > Components: spark >Affects Versions: 2.3 >Reporter: Nikolay Izhikov >Assignee: Nikolay Izhikov >Priority: Major > Labels: bigdata > Fix For: 2.5 > > > Basic support of Spark Data Frame for Ignite implemented in IGNITE-3084. > We need to implement custom spark strategy that can convert whole Spark SQL > query to Ignite SQL Query if query consists of only Ignite tables. > The strategy does nothing if spark query includes not only Ignite tables. > Memsql implementation can be taken as an example - > https://github.com/memsql/memsql-spark-connector -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7077) Spark Data Frame Support. Strategy to convert complete query to Ignite SQL
[ https://issues.apache.org/jira/browse/IGNITE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331926#comment-16331926 ] Vladimir Ozerov commented on IGNITE-7077: - Looks like ticket should be moved to AI 2.5 as 2.4 reached feature complete phase. > Spark Data Frame Support. Strategy to convert complete query to Ignite SQL > -- > > Key: IGNITE-7077 > URL: https://issues.apache.org/jira/browse/IGNITE-7077 > Project: Ignite > Issue Type: New Feature > Components: spark >Affects Versions: 2.3 >Reporter: Nikolay Izhikov >Assignee: Nikolay Izhikov >Priority: Major > Labels: bigdata > Fix For: 2.4 > > > Basic support of Spark Data Frame for Ignite implemented in IGNITE-3084. > We need to implement custom spark strategy that can convert whole Spark SQL > query to Ignite SQL Query if query consists of only Ignite tables. > The strategy does nothing if spark query includes not only Ignite tables. > Memsql implementation can be taken as an example - > https://github.com/memsql/memsql-spark-connector -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7077) Spark Data Frame Support. Strategy to convert complete query to Ignite SQL
[ https://issues.apache.org/jira/browse/IGNITE-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328944#comment-16328944 ] ASF GitHub Bot commented on IGNITE-7077: GitHub user nizhikov opened a pull request: https://github.com/apache/ignite/pull/3397 IGNITE-7077: Implementation of Spark query optimization. You can merge this pull request into a Git repository by running: $ git pull https://github.com/nizhikov/ignite IGNITE-7077 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/3397.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3397 > Spark Data Frame Support. Strategy to convert complete query to Ignite SQL > -- > > Key: IGNITE-7077 > URL: https://issues.apache.org/jira/browse/IGNITE-7077 > Project: Ignite > Issue Type: New Feature > Components: spark >Affects Versions: 2.3 >Reporter: Nikolay Izhikov >Assignee: Nikolay Izhikov >Priority: Major > Labels: bigdata > Fix For: 2.4 > > > Basic support of Spark Data Frame for Ignite implemented in IGNITE-3084. > We need to implement custom spark strategy that can convert whole Spark SQL > query to Ignite SQL Query if query consists of only Ignite tables. > The strategy does nothing if spark query includes not only Ignite tables. > Memsql implementation can be taken as an example - > https://github.com/memsql/memsql-spark-connector -- This message was sent by Atlassian JIRA (v7.6.3#76005)
