[
https://issues.apache.org/jira/browse/HIVE-15179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15655389#comment-15655389
]
Greg Hazel commented on HIVE-15179:
-----------------------------------
Here's a snippet from another failure:
java.lang.IllegalArgumentException: Error: type expected at the position 5618
of
'[ELIDED]metadata:struct<id:string,original_width:int,original_height:int,background_size:string,filter:string,is_done:boo:bigint:boolean:bigint:string:bigint:bigint:bigint:double:string:bigint:string:int:boolean:string:boolean:boolean:string:string:array<struct<[ELIDED]'
but 'boo' is found. at
org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils$TypeInfoParser.expect(TypeInfoUtils.java:348)
at
Again, the [ELIDED] is my edit, but the "boo" fragment is present in the
original error. Something constructing the type string seems to have probably
truncated "boolean". We are constructing a nested
`org.apache.spark.sql.types.StructType`, and passing that straight to
spark.read.schema(struct), so I'm not sure where the type name is serialized to
a string -- but that's probably where the error is.
> Type description limited in size
> --------------------------------
>
> Key: HIVE-15179
> URL: https://issues.apache.org/jira/browse/HIVE-15179
> Project: Hive
> Issue Type: Bug
> Reporter: Greg Hazel
>
> The type for this column is very large, and Parquet seems to give up around
> 4-5k characters. I've "[ELIDED]" most of it, because it is quite large and
> reveals a bunch of schema I am not allowed to share.
> java.lang.IllegalArgumentException: Error: > expected at the position 4136 of
> 'string:string:bigint:bigint:string:string:int:int:string:string:string:int:int:string:string:string:string:struct<post_id:string,[ELIDED]'
> but ':' is found.
> at
> org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils$TypeInfoParser.expect(TypeInfoUtils.java:360)
> at
> org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils$TypeInfoParser.parseType(TypeInfoUtils.java:472)
> at
> org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils$TypeInfoParser.parseType(TypeInfoUtils.java:447)
> at
> org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils$TypeInfoParser.parseType(TypeInfoUtils.java:481)
> at
> org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils$TypeInfoParser.parseType(TypeInfoUtils.java:481)
> at
> org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils$TypeInfoParser.parseType(TypeInfoUtils.java:481)
> at
> org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils$TypeInfoParser.parseTypeInfos(TypeInfoUtils.java:305)
> at
> org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils.getTypeInfosFromTypeString(TypeInfoUtils.java:754)
> at
> org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe.initialize(ParquetHiveSerDe.java:104)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:339)
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:288)
> at
> org.apache.hadoop.hive.ql.metadata.Table.checkValidity(Table.java:194)
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1017)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1.apply(HiveClientImpl.scala:353)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1.apply(HiveClientImpl.scala:351)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:280)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:227)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:226)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:269)
> at
> org.apache.spark.sql.hive.client.HiveClientImpl.getTableOption(HiveClientImpl.scala:351)
> at
> org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$tableExists$1.apply$mcZ$sp(HiveExternalCatalog.scala:228)
> at
> org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$tableExists$1.apply(HiveExternalCatalog.scala:228)
> at
> org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$tableExists$1.apply(HiveExternalCatalog.scala:228)
> at
> org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:72)
> at
> org.apache.spark.sql.hive.HiveExternalCatalog.tableExists(HiveExternalCatalog.scala:227)
> at
> org.apache.spark.sql.catalyst.catalog.SessionCatalog.tableExists(SessionCatalog.scala:456)
> at
> org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:359)
> at
> org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:354)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
