Paul Praet created SPARK-21666:
----------------------------------
Summary: Cannot handle Parquet type FIXED_LEN_BYTE_ARRAY
Key: SPARK-21666
URL: https://issues.apache.org/jira/browse/SPARK-21666
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.2.0
Reporter: Paul Praet
I have a parquet schema that looks like this:
{{ optional group connection {
required fixed_len_byte_array(6) localMacAddress;
required fixed_len_byte_array(6) remoteMacAddress;
}
}}
When I try to load this parquet file in Spark, I get:
Caused by: org.apache.spark.sql.AnalysisException: Illegal Parquet type:
FIXED_LEN_BYTE_ARRAY;
at
org.apache.spark.sql.execution.datasources.parquet.ParquetSchemaConverter.illegalType$1(ParquetSchemaConverter.scala:126)
at
org.apache.spark.sql.execution.datasources.parquet.ParquetSchemaConverter.convertPrimitiveField(ParquetSchemaConverter.scala:193)
at
org.apache.spark.sql.execution.datasources.parquet.ParquetSchemaConverter.convertField(ParquetSchemaConverter.scala:108)
at
org.apache.spark.sql.execution.datasources.parquet.ParquetSchemaConverter$$anonfun$2.apply(ParquetSchemaConverter.scala:90)
at
org.apache.spark.sql.execution.datasources.parquet.ParquetSchemaConverter$$anonfun$2.apply(ParquetSchemaConverter.scala:84)
We are not able to change the schema so this issue prevents us from processing
the data.
Duplicate of https://issues.apache.org/jira/browse/SPARK-2489
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
