Sorabh Hamirwasia created DRILL-6418:
----------------------------------------
Summary: Handle Schema change in Unnest And Lateral for unnest
field / non-unnest field
Key: DRILL-6418
URL: https://issues.apache.org/jira/browse/DRILL-6418
Project: Apache Drill
Issue Type: Improvement
Reporter: Sorabh Hamirwasia
Assignee: Sorabh Hamirwasia
Fix For: 1.14.0
Handling the schema change scenarios for LATERAL and UNNEST when schema change
is observed for non-unnest field and unnest field. It should also handle the
scenario when UNNEST field is Repeated Map Type and schema change happens only
in the children field of Map component. Following issues were found:
1) There were issues found in how scan treats that kind of data where it scans
2 files, one has Map (let say cutomer_order) children type (custKey) as integer
and other has custKey as string type. Then Scan just replaces the ValueVector
from its output container for custKey but in the schema it has 2 fields with
same name but different types.
2) Unnest and Lateral check for schema change across batches based on the
MaterializedField and BatchSchema reference they store. But since it's a
reference any change in pointed value changes their reference as well and hence
comparison always returns true. So for Unnest it actually keeps a clone of the
Materialized Field instead of reference and use that to determine for schema
change. For LATERAL any OK_NEW_SCHEMA from upstream is treated as actual schema
change and setup is done again.
3) Since the MaterializedField is mutable the isEquivalent method checks for
object equality which is removed as that is not correct for mutable objects.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
