[
https://issues.apache.org/jira/browse/SPARK-16926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen updated SPARK-16926:
------------------------------
Assignee: Brian Cho
> Partition columns are present in columns metadata for partition but not table
> -----------------------------------------------------------------------------
>
> Key: SPARK-16926
> URL: https://issues.apache.org/jira/browse/SPARK-16926
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Reporter: Brian Cho
> Assignee: Brian Cho
> Fix For: 2.0.1, 2.1.0
>
>
> A change introduced in SPARK-14388 removes partition columns from the column
> metadata of tables, but not for partitions. This causes TableReader to
> believe that the schema is different and create an unnecessary conversion
> object inspector, taking the else codepath in TableReader below:
> {code}
> val soi = if
> (rawDeser.getObjectInspector.equals(tableDeser.getObjectInspector)) {
> rawDeser.getObjectInspector.asInstanceOf[StructObjectInspector]
> } else {
> ObjectInspectorConverters.getConvertedOI(
> rawDeser.getObjectInspector,
> tableDeser.getObjectInspector).asInstanceOf[StructObjectInspector]
> }
> {code}
> Printing the properties as debug output confirms the difference for the Hive
> table.
> Table properties (tableDesc.getProperties):
> {code}
> 16/08/04 20:36:58 DEBUG HadoopTableReader: columns.types,
> string:bigint:string:bigint:bigint:array<string>
> {code}
> Partition properties (partProps):
> {code}
> 16/08/04 20:36:58 DEBUG HadoopTableReader: columns.types,
> string:bigint:string:bigint:bigint:array<string>:string:string:string
> {code}
> Where the final three string columns are partition columns
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
