[
https://issues.apache.org/jira/browse/PARQUET-390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14990046#comment-14990046
]
Michael Allman commented on PARQUET-390:
----------------------------------------
Hi Ryan,
Correct, I want to test compatibility between the two schemas modulo some basic
primitive type conversions. FWIW, all of our fields are optional so for our use
case that's not an issue.
To answer your question, the object model is the Spark SQL Catalyst data types,
which are converted to parquet types and hive schema.
I think I came up with a tentative solution this morning for testing
compatibility using the catalyst types. I'm going to give that a shot.
Cheers.
> GroupType.union(Type toMerge, boolean strict) does not honor strict parameter
> -----------------------------------------------------------------------------
>
> Key: PARQUET-390
> URL: https://issues.apache.org/jira/browse/PARQUET-390
> Project: Parquet
> Issue Type: Bug
> Components: parquet-mr
> Reporter: Michael Allman
> Labels: newbie, parquet
>
> This is the code as it currently stands in master:
> {code}
> @Override
> protected Type union(Type toMerge, boolean strict) {
> if (toMerge.isPrimitive()) {
> throw new IncompatibleSchemaModificationException("can not merge
> primitive type " + toMerge + " into group type " + this);
> }
> return new GroupType(toMerge.getRepetition(), getName(),
> mergeFields(toMerge.asGroupType()));
> }
> {code}
> Note the call to {{mergeFields}} omits the {{strict}} parameter. I believe
> the code should be:
> {code}
> @Override
> protected Type union(Type toMerge, boolean strict) {
> if (toMerge.isPrimitive()) {
> throw new IncompatibleSchemaModificationException("can not merge
> primitive type " + toMerge + " into group type " + this);
> }
> return new GroupType(toMerge.getRepetition(), getName(),
> mergeFields(toMerge.asGroupType(), strict));
> }
> {code}
> Note the call to {{mergeFields}} includes the {{strict}} parameter.
> I would work on this myself, but I'm having considerable trouble working with
> the codebase (see e.g.
> http://stackoverflow.com/questions/31229445/build-failure-apache-parquet-mr-source-mvn-install-failure).
> Given the (assumed) simplicity of the fix, can a seasoned Parquet
> contributor take this up? Cheers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
