[jira] [Comment Edited] (IGNITE-12141) Ignite Spark Integration Support Schema on Table Write
[ https://issues.apache.org/jira/browse/IGNITE-12141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945983#comment-16945983 ] Manoj G T edited comment on IGNITE-12141 at 10/7/19 3:57 PM: - Dear [~zaleslaw], If I understood correctly as per the implementation of https://issues.apache.org/jira/browse/IGNITE-9228 we can only give schema name to read data, not for write. This feature was implemented during Ignite 2.6 timeline and at that point of time Ignite doesn't allow to create table on any schema other than Public Schema and this is the reason for not supporting "OPTION_SCHEMA" during Overwrite mode. (Reference: [https://github.com/apache/ignite/pull/4551)|https://github.com/apache/ignite/pull/4551] Kindly let me know if this is also addressed so that I can close this bug. was (Author: gtmanoj235): Dear [~zaleslaw], If I understood correctly as per the implementation of https://issues.apache.org/jira/browse/IGNITE-9228 we can only give schema name to read data, not for write. If my understanding is correct, this feature was implemented during Ignite 2.6 timeline and at that point of time Ignite doesn't allow to create table on any schema other than Public Schema and this is the reason for not supporting "OPTION_SCHEMA" during Overwrite mode. (Reference: [https://github.com/apache/ignite/pull/4551)|https://github.com/apache/ignite/pull/4551] Kindly let me know if this is also addressed so that I can close this bug. > Ignite Spark Integration Support Schema on Table Write > -- > > Key: IGNITE-12141 > URL: https://issues.apache.org/jira/browse/IGNITE-12141 > Project: Ignite > Issue Type: Improvement > Components: spark >Affects Versions: 2.7.5 >Reporter: Manoj G T >Assignee: Alexey Zinoviev >Priority: Critical > Fix For: 2.8 > > Original Estimate: 4h > Remaining Estimate: 4h > > Ignite 2.6 doesn't allow to create table on any schema other than Public > Schema and this is the reason for not supporting "OPTION_SCHEMA" during > Overwrite mode. Now that Ignite supports to create the table in any given > schema it will be great if we can incorporate the changes to support > "OPTION_SCHEMA" during Overwrite mode and make it available as part of next > Ignite release. > > +Related Issue:+ > [https://stackoverflow.com/questions/57782033/apache-ignite-spark-integration-not-working-with-schema-name] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-12141) Ignite Spark Integration Support Schema on Table Write
[ https://issues.apache.org/jira/browse/IGNITE-12141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945983#comment-16945983 ] Manoj G T commented on IGNITE-12141: Dear [~zaleslaw], If I understood correctly as per the implementation of https://issues.apache.org/jira/browse/IGNITE-9228 we can only give schema name to read data, not for write. If my understanding is correct, this feature was implemented during Ignite 2.6 timeline and at that point of time Ignite doesn't allow to create table on any schema other than Public Schema and this is the reason for not supporting "OPTION_SCHEMA" during Overwrite mode. (Reference: [https://github.com/apache/ignite/pull/4551)|https://github.com/apache/ignite/pull/4551] Kindly let me know if this is also addressed so that I can close this bug. > Ignite Spark Integration Support Schema on Table Write > -- > > Key: IGNITE-12141 > URL: https://issues.apache.org/jira/browse/IGNITE-12141 > Project: Ignite > Issue Type: Improvement > Components: spark >Affects Versions: 2.7.5 >Reporter: Manoj G T >Assignee: Alexey Zinoviev >Priority: Critical > Fix For: 2.8 > > Original Estimate: 4h > Remaining Estimate: 4h > > Ignite 2.6 doesn't allow to create table on any schema other than Public > Schema and this is the reason for not supporting "OPTION_SCHEMA" during > Overwrite mode. Now that Ignite supports to create the table in any given > schema it will be great if we can incorporate the changes to support > "OPTION_SCHEMA" during Overwrite mode and make it available as part of next > Ignite release. > > +Related Issue:+ > [https://stackoverflow.com/questions/57782033/apache-ignite-spark-integration-not-working-with-schema-name] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-9228) Spark SQL Table Schema Specification
[ https://issues.apache.org/jira/browse/IGNITE-9228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923920#comment-16923920 ] Manoj G T commented on IGNITE-9228: --- Thanks, [~NIzhikov]. I have converted my ticket from bug to an Improvement regarding the same https://issues.apache.org/jira/browse/IGNITE-12141 and added you as the reviewer. My Colleague or I will take this ticket early next week. > Spark SQL Table Schema Specification > > > Key: IGNITE-9228 > URL: https://issues.apache.org/jira/browse/IGNITE-9228 > Project: Ignite > Issue Type: Improvement > Components: spark >Affects Versions: 2.6 >Reporter: Stuart Macdonald >Assignee: Stuart Macdonald >Priority: Major > Fix For: 2.8 > > > The Ignite Spark SQL interface currently takes just “table name” as a > parameter which it uses to supply a Spark dataset with data from the > underlying Ignite SQL table with that name. > To do this it loops through each cache and finds the first one with the > given table name [1]. This causes issues if there are multiple tables > registered in different schema with the same table name as you can only > access one of those from Spark. We could either: > 1. Pass an extra parameter through the Ignite Spark data source which > optionally specifies the schema name. > 2. Support namespacing in the existing table name parameter, ie > “schemaName.tableName” > [1 > ]https://github.com/apache/ignite/blob/ca973ad99c6112160a305df05be9458e29f88307/modules/spark/src/main/scala/org/apache/ignite/spark/impl/package.scala#L119 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (IGNITE-12141) Ignite Spark Integration Support Schema on Table Write
[ https://issues.apache.org/jira/browse/IGNITE-12141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manoj G T updated IGNITE-12141: --- Fix Version/s: (was: 2.7.6) 2.8 Reviewer: Nikolay Izhikov Description: Ignite 2.6 doesn't allow to create table on any schema other than Public Schema and this is the reason for not supporting "OPTION_SCHEMA" during Overwrite mode. Now that Ignite supports to create the table in any given schema it will be great if we can incorporate the changes to support "OPTION_SCHEMA" during Overwrite mode and make it available as part of next Ignite release. +Related Issue:+ [https://stackoverflow.com/questions/57782033/apache-ignite-spark-integration-not-working-with-schema-name] was: Users can't able to specify schema when trying to persist Spark DF to Ignite. [https://stackoverflow.com/questions/57782033/apache-ignite-spark-integration-not-working-with-schema-name] Issue Type: Improvement (was: Bug) Summary: Ignite Spark Integration Support Schema on Table Write (was: Ignite Spark Integration not working with Schema Name) Remaining Estimate: 4h Original Estimate: 4h > Ignite Spark Integration Support Schema on Table Write > -- > > Key: IGNITE-12141 > URL: https://issues.apache.org/jira/browse/IGNITE-12141 > Project: Ignite > Issue Type: Improvement > Components: spark >Affects Versions: 2.7.5 >Reporter: Manoj G T >Priority: Critical > Fix For: 2.8 > > Original Estimate: 4h > Remaining Estimate: 4h > > Ignite 2.6 doesn't allow to create table on any schema other than Public > Schema and this is the reason for not supporting "OPTION_SCHEMA" during > Overwrite mode. Now that Ignite supports to create the table in any given > schema it will be great if we can incorporate the changes to support > "OPTION_SCHEMA" during Overwrite mode and make it available as part of next > Ignite release. > > +Related Issue:+ > [https://stackoverflow.com/questions/57782033/apache-ignite-spark-integration-not-working-with-schema-name] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (IGNITE-9228) Spark SQL Table Schema Specification
[ https://issues.apache.org/jira/browse/IGNITE-9228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923107#comment-16923107 ] Manoj G T commented on IGNITE-9228: --- [~NIzhikov] [~stuartmacd] I have read the code review comments given in [https://github.com/apache/ignite/pull/4551]. If my understanding is correct, this feature was implemented during Ignite 2.6 timeline and at that point of time Ignite doesn't allow to create table on any schema other than Public Schema and this is the reason for not supporting "OPTION_SCHEMA" during Overwrite mode. Now that Ignite supports to create the table in any given schema it will be great if we can incorporate the changes to support "OPTION_SCHEMA" during Overwrite mode and make it available as part of next Ignite release. Kindly share your thoughts on this. > Spark SQL Table Schema Specification > > > Key: IGNITE-9228 > URL: https://issues.apache.org/jira/browse/IGNITE-9228 > Project: Ignite > Issue Type: Improvement > Components: spark >Affects Versions: 2.6 >Reporter: Stuart Macdonald >Assignee: Stuart Macdonald >Priority: Major > Fix For: 2.8 > > > The Ignite Spark SQL interface currently takes just “table name” as a > parameter which it uses to supply a Spark dataset with data from the > underlying Ignite SQL table with that name. > To do this it loops through each cache and finds the first one with the > given table name [1]. This causes issues if there are multiple tables > registered in different schema with the same table name as you can only > access one of those from Spark. We could either: > 1. Pass an extra parameter through the Ignite Spark data source which > optionally specifies the schema name. > 2. Support namespacing in the existing table name parameter, ie > “schemaName.tableName” > [1 > ]https://github.com/apache/ignite/blob/ca973ad99c6112160a305df05be9458e29f88307/modules/spark/src/main/scala/org/apache/ignite/spark/impl/package.scala#L119 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (IGNITE-9228) Spark SQL Table Schema Specification
[ https://issues.apache.org/jira/browse/IGNITE-9228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923059#comment-16923059 ] Manoj G T commented on IGNITE-9228: --- [~NIzhikov] [~stuartmacd] I have gone through the implementation of supporting schema name Ignite Spark SQL and it's only having support to schema name to read data, not for write. Any specific reason for the same? Is it in the future roadmap? > Spark SQL Table Schema Specification > > > Key: IGNITE-9228 > URL: https://issues.apache.org/jira/browse/IGNITE-9228 > Project: Ignite > Issue Type: Improvement > Components: spark >Affects Versions: 2.6 >Reporter: Stuart Macdonald >Assignee: Stuart Macdonald >Priority: Major > Fix For: 2.8 > > > The Ignite Spark SQL interface currently takes just “table name” as a > parameter which it uses to supply a Spark dataset with data from the > underlying Ignite SQL table with that name. > To do this it loops through each cache and finds the first one with the > given table name [1]. This causes issues if there are multiple tables > registered in different schema with the same table name as you can only > access one of those from Spark. We could either: > 1. Pass an extra parameter through the Ignite Spark data source which > optionally specifies the schema name. > 2. Support namespacing in the existing table name parameter, ie > “schemaName.tableName” > [1 > ]https://github.com/apache/ignite/blob/ca973ad99c6112160a305df05be9458e29f88307/modules/spark/src/main/scala/org/apache/ignite/spark/impl/package.scala#L119 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (IGNITE-12141) Ignite Spark Integration not working with Schema Name
[ https://issues.apache.org/jira/browse/IGNITE-12141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manoj G T updated IGNITE-12141: --- Ignite Flags: Docs Required (was: Docs Required,Release Notes Required) > Ignite Spark Integration not working with Schema Name > - > > Key: IGNITE-12141 > URL: https://issues.apache.org/jira/browse/IGNITE-12141 > Project: Ignite > Issue Type: Bug > Components: spark >Affects Versions: 2.7.5 >Reporter: Manoj G T >Priority: Critical > > Users can't able to specify schema when trying to persist Spark DF to Ignite. > > [https://stackoverflow.com/questions/57782033/apache-ignite-spark-integration-not-working-with-schema-name] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (IGNITE-12141) Ignite Spark Integration not working with Schema Name
[ https://issues.apache.org/jira/browse/IGNITE-12141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manoj G T updated IGNITE-12141: --- Fix Version/s: 2.7.6 > Ignite Spark Integration not working with Schema Name > - > > Key: IGNITE-12141 > URL: https://issues.apache.org/jira/browse/IGNITE-12141 > Project: Ignite > Issue Type: Bug > Components: spark >Affects Versions: 2.7.5 >Reporter: Manoj G T >Priority: Critical > Fix For: 2.7.6 > > > Users can't able to specify schema when trying to persist Spark DF to Ignite. > > [https://stackoverflow.com/questions/57782033/apache-ignite-spark-integration-not-working-with-schema-name] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (IGNITE-12141) Ignite Spark Integration not working with Schema Name
Manoj G T created IGNITE-12141: -- Summary: Ignite Spark Integration not working with Schema Name Key: IGNITE-12141 URL: https://issues.apache.org/jira/browse/IGNITE-12141 Project: Ignite Issue Type: Bug Components: spark Affects Versions: 2.7.5 Reporter: Manoj G T Users can't able to specify schema when trying to persist Spark DF to Ignite. [https://stackoverflow.com/questions/57782033/apache-ignite-spark-integration-not-working-with-schema-name] -- This message was sent by Atlassian Jira (v8.3.2#803003)
