Ángel Álvarez created SPARK-9685:
------------------------------------
Summary: "Unsupported dataType: char(X)" in Hive
Key: SPARK-9685
URL: https://issues.apache.org/jira/browse/SPARK-9685
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 1.5.0
Reporter: Ángel Álvarez
I'm getting the following error when I try to read a Hive table with char(X)
fields:
{code}
15/08/06 11:38:51 INFO parse.ParseDriver: Parse Completed
org.apache.spark.sql.types.DataTypeException: Unsupported dataType: char(8). If
you have a struct and a field name of it has any special characters, please use
backticks (`) to quote that field name, e.g. `x+y`. Please note that backtick
itself is not supported in a field name.
at
org.apache.spark.sql.types.DataTypeParser$class.toDataType(DataTypeParser.scala:95)
at
org.apache.spark.sql.types.DataTypeParser$$anon$1.toDataType(DataTypeParser.scala:107)
at
org.apache.spark.sql.types.DataTypeParser$.parse(DataTypeParser.scala:111)
at
org.apache.spark.sql.hive.HiveMetastoreTypes$.toDataType(HiveMetastoreCatalog.scala:769)
at
org.apache.spark.sql.hive.MetastoreRelation$SchemaAttribute.toAttribute(HiveMetastoreCatalog.scala:742)
at
org.apache.spark.sql.hive.MetastoreRelation$$anonfun$44.apply(HiveMetastoreCatalog.scala:752)
at
org.apache.spark.sql.hive.MetastoreRelation$$anonfun$44.apply(HiveMetastoreCatalog.scala:752)
{code}
It seems there is no "char" DataType defined in the DataTypeParser class
{code}
protected lazy val primitiveType: Parser[DataType] =
"(?i)string".r ^^^ StringType |
"(?i)float".r ^^^ FloatType |
"(?i)(?:int|integer)".r ^^^ IntegerType |
"(?i)tinyint".r ^^^ ByteType |
"(?i)smallint".r ^^^ ShortType |
"(?i)double".r ^^^ DoubleType |
"(?i)(?:bigint|long)".r ^^^ LongType |
"(?i)binary".r ^^^ BinaryType |
"(?i)boolean".r ^^^ BooleanType |
fixedDecimalType |
"(?i)decimal".r ^^^ DecimalType.USER_DEFAULT |
"(?i)date".r ^^^ DateType |
"(?i)timestamp".r ^^^ TimestampType |
varchar
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
