[
https://issues.apache.org/jira/browse/IGNITE-10314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ray updated IGNITE-10314:
-------------------------
Description:
When user performs add/remove column in DDL, Spark will get the old/wrong
schema.
Analyse
Currently Spark data frame API relies on QueryEntity to construct schema, but
QueryEntity in QuerySchema is a local copy of the original QueryEntity, so the
original QueryEntity is not updated when modification happens.
Solution
Get the latest schema using JDBC thin driver's column metadata call, then
update fields in QueryEntity.
was:
When user performs add/remove column in DDL, Spark will get the old/wrong
schema.
Analyse
Currently Spark data frame API relies on QueryEntity to construct schema, but
QueryEntity in QuerySchema is a local copy of the original QueryEntity, so the
original QueryEntity is not updated when modification happens.
Solution
Get the schema using sql, get rid of QueryEntity.
> Spark dataframe will get wrong schema if user executes add/drop column DDL
> --------------------------------------------------------------------------
>
> Key: IGNITE-10314
> URL: https://issues.apache.org/jira/browse/IGNITE-10314
> Project: Ignite
> Issue Type: Bug
> Components: spark
> Affects Versions: 2.3, 2.4, 2.5, 2.6, 2.7
> Reporter: Ray
> Assignee: Ray
> Priority: Critical
> Fix For: 2.8
>
>
> When user performs add/remove column in DDL, Spark will get the old/wrong
> schema.
>
> Analyse
> Currently Spark data frame API relies on QueryEntity to construct schema, but
> QueryEntity in QuerySchema is a local copy of the original QueryEntity, so
> the original QueryEntity is not updated when modification happens.
>
> Solution
> Get the latest schema using JDBC thin driver's column metadata call, then
> update fields in QueryEntity.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
