[ https://issues.apache.org/jira/browse/SPARK-11202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15006897#comment-15006897 ]
F Jimenez commented on SPARK-11202: ----------------------------------- I have noticed the following commit that may have solved the problem https://github.com/apache/spark/commit/02149ff08eed3745086589a047adbce9a580389f > Unsupported dataType > -------------------- > > Key: SPARK-11202 > URL: https://issues.apache.org/jira/browse/SPARK-11202 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 1.5.0 > Reporter: whc > > I read data from oracle and save as parquet ,then get the following error: > java.lang.IllegalArgumentException: Unsupported dataType: > {"type":"struct","fields":[{"name":"DOMAIN_NAME","type":"string","nullable":true,"metadata":{"name":"DOMAIN_NAME"}},{"name":"DOMAIN_ID","type":"decimal(0,-127)","nullable":true,"metadata":{"name":"DOMAIN_ID"}}]}, > [1.1] failure: `TimestampType' expected but `{' found > {"type":"struct","fields":[{"name":"DOMAIN_NAME","type":"string","nullable":true,"metadata":{"name":"DOMAIN_NAME"}},{"name":"DOMAIN_ID","type":"decimal(0,-127)","nullable":true,"metadata":{"name":"DOMAIN_ID"}}]} > ^ > at > org.apache.spark.sql.types.DataType$CaseClassStringParser$.apply(DataType.scala:245) > at > org.apache.spark.sql.types.DataType$.fromCaseClassString(DataType.scala:102) > at > org.apache.spark.sql.execution.datasources.parquet.ParquetTypesConverter$$anonfun$3.apply(ParquetTypesConverter.scala:62) > at > org.apache.spark.sql.execution.datasources.parquet.ParquetTypesConverter$$anonfun$3.apply(ParquetTypesConverter.scala:62) > at scala.util.Try.getOrElse(Try.scala:77) > at > org.apache.spark.sql.execution.datasources.parquet.ParquetTypesConverter$.convertFromString(ParquetTypesConverter.scala:62) > at > org.apache.spark.sql.execution.datasources.parquet.RowWriteSupport.init(ParquetTableSupport.scala:51) > at > org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:288) > at > org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:262) > at > org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.<init>(ParquetRelation.scala:94) > at > org.apache.spark.sql.execution.datasources.parquet.ParquetRelation$$anon$3.newInstance(ParquetRelation.scala:272) > at > org.apache.spark.sql.execution.datasources.DefaultWriterContainer.writeRows(WriterContainer.scala:234) > at > org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1$$anonfun$apply$mcV$sp$3.apply(InsertIntoHadoopFsRelation.scala:150) > at > org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1$$anonfun$apply$mcV$sp$3.apply(InsertIntoHadoopFsRelation.scala:150) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) > at org.apache.spark.scheduler.Task.run(Task.scala:88) > at > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > I checked the type but do not have Timestamp or Date type in oracle > my oracle table like this: > create table DW_DOMAIN > ( > domain_id NUMBER, > cityid NUMBER, > domain_type NUMBER, > domain_name VARCHAR2(80) > ) > and my code like this: > Map<String, String> options = new HashMap<String, String>(); > options.put("url", url); > options.put("driver", driver); > options.put("user", user); > options.put("password", password); > options.put("dbtable", "(select DOMAIN_NAME,DOMAIN_ID from > dw_domain ) t"); > DataFrame df = this.sqlContext.read().format("jdbc").options(options ) > .load(); > df.write().mode(SaveMode.Append) > > .parquet("hdfs://cluster1:8020/database/count_domain/"); > if add "to_char(DOMAIN_ID)",that can get correct result. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org