Github user maropu commented on the issue:
https://github.com/apache/spark/pull/16030
This is not a bug in `VectorizedParquetReader ` as @liancheng said, and
the root cause is that wrongly inferred types are passed into the reader in
`VectorizedParquetReader#initBatch`
https://github.com/apache/spark/blob/master/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java#L176
yea, I know this functionality is helpful for skilful users, and, on the
other hand, newbies easily could break query results via this interface as
@brkyvz said. Therefore, in case of that we allow data schema to contain (part
of) the partition columns, IMO it'd be better to alert that risk to users via
`logWarning` or something.
BTW this is the original fix of this bug (Sorry, but I wrongly overrode and
remove this commit ):
https://github.com/apache/spark/compare/master...maropu:SPARK-18108-2. The fix
of this commit is to fill in correct data types when data schema contains (part
of) the partition columns.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
