[
https://issues.apache.org/jira/browse/DRILL-6418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sorabh Hamirwasia updated DRILL-6418:
-------------------------------------
Labels: ready-to-commit (was: )
> Handle Schema change in Unnest And Lateral for unnest field / non-unnest field
> ------------------------------------------------------------------------------
>
> Key: DRILL-6418
> URL: https://issues.apache.org/jira/browse/DRILL-6418
> Project: Apache Drill
> Issue Type: Improvement
> Reporter: Sorabh Hamirwasia
> Assignee: Sorabh Hamirwasia
> Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> Handling the schema change scenarios for LATERAL and UNNEST when schema
> change is observed for non-unnest field and unnest field. It should also
> handle the scenario when UNNEST field is Repeated Map Type and schema change
> happens only in the children field of Map component. Following issues were
> found:
> 1) There were issues found in how scan treats that kind of data where it
> scans 2 files, one has Map (let say cutomer_order) children type (custKey) as
> integer and other has custKey as string type. Then Scan just replaces the
> ValueVector from its output container for custKey but in the schema it has 2
> fields with same name but different types.
> 2) Unnest and Lateral check for schema change across batches based on the
> MaterializedField and BatchSchema reference they store. But since it's a
> reference any change in pointed value changes their reference as well and
> hence comparison always returns true. So for Unnest it actually keeps a clone
> of the Materialized Field instead of reference and use that to determine for
> schema change. For LATERAL any OK_NEW_SCHEMA from upstream is treated as
> actual schema change and setup is done again.
> 3) Since the MaterializedField is mutable the isEquivalent method checks for
> object equality which is removed as that is not correct for mutable objects.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
