[ https://issues.apache.org/jira/browse/SPARK-23230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xiao Li resolved SPARK-23230. ----------------------------- Resolution: Fixed Assignee: dzcxzl Fix Version/s: 2.3.0 > When hive.default.fileformat is other kinds of file types, create textfile > table cause a serde error > ---------------------------------------------------------------------------------------------------- > > Key: SPARK-23230 > URL: https://issues.apache.org/jira/browse/SPARK-23230 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.0.0, 2.0.1, 2.0.2, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.2.1 > Reporter: dzcxzl > Assignee: dzcxzl > Priority: Minor > Fix For: 2.3.0 > > > When hive.default.fileformat is other kinds of file types, create textfile > table cause a serde error. > We should take the default type of textfile and sequencefile both as > org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe. > {code:java} > set hive.default.fileformat=orc; > create table tbl( i string ) stored as textfile; > desc formatted tbl; > Serde Library org.apache.hadoop.hive.ql.io.orc.OrcSerde > InputFormat org.apache.hadoop.mapred.TextInputFormat > OutputFormat org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat{code} > > {code:java} > set hive.default.fileformat=orc; > create table tbl stored as textfile > as > select 1 > {code} > {{It failed because it used the wrong SERDE}} > {code:java} > Caused by: java.lang.ClassCastException: > org.apache.hadoop.hive.ql.io.orc.OrcSerde$OrcSerdeRow cannot be cast to > org.apache.hadoop.io.BytesWritable > at > org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat$1.write(HiveIgnoreKeyTextOutputFormat.java:91) > at > org.apache.spark.sql.hive.execution.HiveOutputWriter.write(HiveFileFormat.scala:149) > at > org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.execute(FileFormatWriter.scala:327) > at > org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:258) > at > org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:256) > at > org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1375) > at > org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:261) > ... 16 more > {code} > -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org