[
https://issues.apache.org/jira/browse/DRILL-5828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16186815#comment-16186815
]
Aman Sinha commented on DRILL-5828:
-----------------------------------
The second case with different order of the columns is not a real schema change
from Drill perspective because the lookup at run time is done based on the
column names, not positions. The top level Project operator does need to
produce the desired order but intermediate operators in the pipeline can
operate on the name as defined by MaterializedField.
> RecordBatchLoader permutes column order
> ---------------------------------------
>
> Key: DRILL-5828
> URL: https://issues.apache.org/jira/browse/DRILL-5828
> Project: Apache Drill
> Issue Type: Bug
> Affects Versions: 1.11.0
> Reporter: Paul Rogers
> Priority: Minor
>
> The {{RecordBatchLoader}} class deserializes batches and checks for schema
> changes. As part of investigating DRILL-5826, it seems that
> {{RecordBatchLoader}} detects schema changes as follows:
> * If two batches have the same column in the same order, no schema change
> occurs. (Fine)
> * If batch A has schema (a, b) while batch B has (b, a), then no schema
> change occurs. (Fine)
> But, in the case of permutated columns (second case above), the
> {{RecordBatchLoader}} returns the column order of the second batch, though it
> says that no schema change has occurred.
> That is, {{RecordBatchLoader}} says that the schema has not changed, but the
> actual schema has changed (column order changed.)
> This is a potential problem: if a downstream batch counts on the same column
> order, then that assumption is violated by the behavior described above.
> Correct behavior would be to coerce the second batch to match the schema of
> the first batch, if the {{RecordBatchLoader}} indicates that no schema change
> occurred.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
