[
https://issues.apache.org/jira/browse/HIVE-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15371651#comment-15371651
]
Owen O'Malley edited comment on HIVE-13974 at 7/11/16 10:07 PM:
----------------------------------------------------------------
[~owen.omalley] Thanks for looking at this.
No, the semantics of sameCategoryAndAttributes is different than equals. The
TypeDescription.equals method compares (type) id and maximumId which does not
work when there is an interior STRUCT column with a different number of
columns. It makes it seem like a type conversion is needed when one is not
needed and other parts of the code throw exceptions complaining "no need to
convert a STRING to a STRING".
There are 3 kinds of schema not 2. Part of the problem I'm trying to solve is
the ambiguity at different parts of the code as to which schema is being used.
It is the one being returned by the input file format, is it the schema being
fed back to the ORC raw merger that included ACID columns, or is it the
unconverted file schema. I don't care what the first 2 schemas are called as
long as the names are distinct. Maybe the names could be reader,
internalReader, and file.
About ORC-54 -- it is not practical right now in terms of time. We have got to
get Erie out the door. We have so little runway left. I've had 10+ JIRAs for
weeks. Whenever I knock some down more appear. Also, there really needs to be
a parallel HIVE JIRA for it and we must make sure name mapping is fully
supported for HIVE. Given how *difficult* Schema Evolution has been I simply
don't believe it will *just work* with ORC only unit tests.
FYI [~hagleitn] [~ekoifman]
was (Author: mmccline):
{quote}
No, the semantics of sameCategoryAndAttributes is different than equals.
{quote}
*Sigh* Ok, I forgot that I had only fixed that on the ORC side of the world as
part of ORC-53. Hive will get that as soon as HIVE-14007 goes in (or is a
negative patch of 2MB "going out"?). In any case, do not add the new method.
ORC-53's impact on orc-core is pretty small outside of TypeDescription. Would
you like a back port of that patch?
{quote}
There are 3 kinds of schema not 2.
{quote}
Ugh. That seems unnecessary. The 'file' schema is pretty clear. The 'reader'
schema is the one that the user asked for. I don't think we need anything else.
{quote}
About ORC-54 -- it is not practical right now in terms of time.
{quote}
ORC-54 is closer to going in. It has unit tests and I believe handles this as a
sub-case. I'm trying to figure out what we gain out of the HIVE-13974 patch.
{quote}
Also, there really needs to be a parallel HIVE JIRA for it and we must make
sure name mapping is fully supported for HIVE.
{quote}
Uh no. The Hive ORC code is about to disappear with HIVE-14007. Continuing to
maintain two versions of ORC with a forked code base is a bad thing.
{code}
Given how *difficult* Schema Evolution has been I simply don't believe it will
*just work* with ORC only unit tests.
{code}
That is not an excuse. Unit tests are MUCH more likely to be correct because
the errors aren't hidden under layers of the execution engine.
> ORC Schema Evolution doesn't support add columns to non-last STRUCT columns
> ---------------------------------------------------------------------------
>
> Key: HIVE-13974
> URL: https://issues.apache.org/jira/browse/HIVE-13974
> Project: Hive
> Issue Type: Bug
> Components: Hive, ORC, Transactions
> Affects Versions: 1.3.0, 2.1.0, 2.2.0
> Reporter: Matt McCline
> Assignee: Matt McCline
> Priority: Blocker
> Attachments: HIVE-13974.01.patch, HIVE-13974.02.patch,
> HIVE-13974.03.patch, HIVE-13974.04.patch, HIVE-13974.05.WIP.patch,
> HIVE-13974.06.patch, HIVE-13974.07.patch, HIVE-13974.08.patch,
> HIVE-13974.09.patch, HIVE-13974.091.patch
>
>
> Currently, the included columns are based on the fileSchema and not the
> readerSchema which doesn't work for adding columns to non-last STRUCT data
> type columns.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
