[
https://issues.apache.org/jira/browse/SPARK-14330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-14330:
---------------------------------
Labels: bulk-closed (was: )
> Spark SQL does not infer correct type for joda DateTime
> -------------------------------------------------------
>
> Key: SPARK-14330
> URL: https://issues.apache.org/jira/browse/SPARK-14330
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 1.6.0
> Reporter: Michel Lemay
> Priority: Minor
> Labels: bulk-closed
>
> Creating a Dataframe on a pojo with joda DateTime:
> {code}
> val df = sqlContext.createDataFrame(rdd, classOf[MyJavaClassWithJodaDateTime])
> df.printSchema
> {code}
> Digging, I find that JavaTypeInference yields unexpected results:
> {code}
> val (dataType, _) =
> org.apache.spark.sql.catalyst.JavaTypeInference.inferDataType(classOf[org.joda.time.DateTime])
> dataType.asInstanceOf[org.apache.spark.sql.types.StructType].fields.foreach(println)
> StructField(afterNow,BooleanType,false)
> StructField(beforeNow,BooleanType,false)
> StructField(centuryOfEra,IntegerType,false)
> StructField(chronology,StructType(StructField(zone,StructType(StructField(ID,StringType,true),
> StructField(fixed,BooleanType,false)),true)),t
> StructField(dayOfMonth,IntegerType,false)
> StructField(dayOfWeek,IntegerType,false)
> StructField(dayOfYear,IntegerType,false)
> StructField(equalNow,BooleanType,false)
> StructField(era,IntegerType,false)
> StructField(hourOfDay,IntegerType,false)
> StructField(millis,LongType,false)
> StructField(millisOfDay,IntegerType,false)
> StructField(millisOfSecond,IntegerType,false)
> StructField(minuteOfDay,IntegerType,false)
> StructField(minuteOfHour,IntegerType,false)
> StructField(monthOfYear,IntegerType,false)
> StructField(secondOfDay,IntegerType,false)
> StructField(secondOfMinute,IntegerType,false)
> StructField(weekOfWeekyear,IntegerType,false)
> StructField(weekyear,IntegerType,false)
> StructField(year,IntegerType,false)
> StructField(yearOfCentury,IntegerType,false)
> StructField(yearOfEra,IntegerType,false)
> StructField(zone,StructType(StructField(ID,StringType,true),
> StructField(fixed,BooleanType,false)),true)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
