[
https://issues.apache.org/jira/browse/PARQUET-390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14990028#comment-14990028
]
Ryan Blue commented on PARQUET-390:
-----------------------------------
Good to know. It sounds like what you're after isn't a union of the schemas but
a test to see if evolution from one to the other is valid or supported. That's
not an area we do particularly well right now and I don't think that this union
function would help. I guess it union(new, old) == new then you know it works.
I think we need to come up with a set of schema evolution rules and tests that
validate them. Some things we know probably won't work, like type coercion
(especially when using logical types with different primitive representations).
Other things like changing a required field to an optional field work, but I
don't think union handles this case correctly.
What object model are you using? Hive?
> GroupType.union(Type toMerge, boolean strict) does not honor strict parameter
> -----------------------------------------------------------------------------
>
> Key: PARQUET-390
> URL: https://issues.apache.org/jira/browse/PARQUET-390
> Project: Parquet
> Issue Type: Bug
> Components: parquet-mr
> Reporter: Michael Allman
> Labels: newbie, parquet
>
> This is the code as it currently stands in master:
> {code}
> @Override
> protected Type union(Type toMerge, boolean strict) {
> if (toMerge.isPrimitive()) {
> throw new IncompatibleSchemaModificationException("can not merge
> primitive type " + toMerge + " into group type " + this);
> }
> return new GroupType(toMerge.getRepetition(), getName(),
> mergeFields(toMerge.asGroupType()));
> }
> {code}
> Note the call to {{mergeFields}} omits the {{strict}} parameter. I believe
> the code should be:
> {code}
> @Override
> protected Type union(Type toMerge, boolean strict) {
> if (toMerge.isPrimitive()) {
> throw new IncompatibleSchemaModificationException("can not merge
> primitive type " + toMerge + " into group type " + this);
> }
> return new GroupType(toMerge.getRepetition(), getName(),
> mergeFields(toMerge.asGroupType(), strict));
> }
> {code}
> Note the call to {{mergeFields}} includes the {{strict}} parameter.
> I would work on this myself, but I'm having considerable trouble working with
> the codebase (see e.g.
> http://stackoverflow.com/questions/31229445/build-failure-apache-parquet-mr-source-mvn-install-failure).
> Given the (assumed) simplicity of the fix, can a seasoned Parquet
> contributor take this up? Cheers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
