[
https://issues.apache.org/jira/browse/IGNITE-10314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ray Liu updated IGNITE-10314:
-----------------------------
Description:
When user performs add/remove column in DDL, Spark will get the old/wrong
schema.
Analyse
Currently Spark data frame API relies on QueryEntity to construct schema, but
QueryEntity in QuerySchema is a local copy of the original QueryEntity, so the
original QueryEntity is not updated when modification happens.
Solution
Use GridQueryTypeDescriptor to replace QueryEntity
was:
When user performs add/remove column in DDL, Spark will get the old/wrong
schema.
Analyse
Currently Spark data frame API relies on QueryEntity to construct schema, but
QueryEntity in QuerySchema is a local copy of the original QueryEntity, so the
original QueryEntity is not updated when modification happens.
Solution
Get the latest schema using JDBC thin driver's column metadata call, then
update fields in QueryEntity.
> Spark dataframe will get wrong schema if user executes add/drop column DDL
> --------------------------------------------------------------------------
>
> Key: IGNITE-10314
> URL: https://issues.apache.org/jira/browse/IGNITE-10314
> Project: Ignite
> Issue Type: Bug
> Components: spark
> Affects Versions: 2.3, 2.4, 2.5, 2.6, 2.7
> Reporter: Ray Liu
> Assignee: Ray Liu
> Priority: Critical
> Fix For: 2.8
>
>
> When user performs add/remove column in DDL, Spark will get the old/wrong
> schema.
>
> Analyse
> Currently Spark data frame API relies on QueryEntity to construct schema, but
> QueryEntity in QuerySchema is a local copy of the original QueryEntity, so
> the original QueryEntity is not updated when modification happens.
>
> Solution
> Use GridQueryTypeDescriptor to replace QueryEntity
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
