Seems I made a mistake… From: Wang, Daoyuan Sent: Tuesday, October 21, 2014 10:35 AM To: 'Yin Huai' Cc: Michael Armbrust; tridib; u...@spark.incubator.apache.org Subject: RE: spark sql: timestamp in json - fails
I got that, it is in JsonRDD.java of `typeOfPrimitiveValues`. I’ll fix that together. Thanks, Daoyuan From: Yin Huai [mailto:huaiyin....@gmail.com] Sent: Tuesday, October 21, 2014 10:13 AM To: Wang, Daoyuan Cc: Michael Armbrust; tridib; u...@spark.incubator.apache.org<mailto:u...@spark.incubator.apache.org> Subject: Re: spark sql: timestamp in json - fails Seems the second approach does not go through applySchema. So, I was wondering if there is an issue related to our JSON apis in Java. On Mon, Oct 20, 2014 at 10:04 PM, Wang, Daoyuan <daoyuan.w...@intel.com<mailto:daoyuan.w...@intel.com>> wrote: I think this has something to do with my recent work at https://issues.apache.org/jira/browse/SPARK-4003 You can check PR https://github.com/apache/spark/pull/2850 . Thanks, Daoyuan From: Yin Huai [mailto:huaiyin....@gmail.com<mailto:huaiyin....@gmail.com>] Sent: Tuesday, October 21, 2014 10:00 AM To: Michael Armbrust Cc: tridib; u...@spark.incubator.apache.org<mailto:u...@spark.incubator.apache.org> Subject: Re: spark sql: timestamp in json - fails Hi Tridib, For the second approach, can you attach the complete stack trace? Thanks, Yin On Mon, Oct 20, 2014 at 8:24 PM, Michael Armbrust <mich...@databricks.com<mailto:mich...@databricks.com>> wrote: I think you are running into a bug that will be fixed by this PR: https://github.com/apache/spark/pull/2850 On Mon, Oct 20, 2014 at 4:34 PM, tridib <tridib.sama...@live.com<mailto:tridib.sama...@live.com>> wrote: Hello Experts, After repeated attempt I am unable to run query on map json date string. I tried two approaches: *** Approach 1 *** created a Bean class with timespamp field. When I try to run it I get scala.MatchError: class java.sql.Timestamp (of class java.lang.Class). Here is the code: import java.sql.Timestamp; public class ComplexClaim { private Timestamp timestamp; public Timestamp getTimestamp() { return timestamp; } public void setTimestamp(Timestamp timestamp) { this.timestamp = timestamp; } } JavaSparkContext ctx = getCtx(sc); JavaSQLContext sqlCtx = getSqlCtx(getCtx(sc)); String path = "/hdd/spark/test.json"; JavaSchemaRDD test = sqlCtx.applySchema(ctx.textFile(path) , ComplexClaim.class); sqlCtx.registerRDDAsTable(test, "test"); execSql(sqlCtx, "select * from test", 1); *** Approach 2 *** Created a StructType to map the date field. I got scala.MatchError: TimestampType (of class org.apache.spark.sql.catalyst.types.TimestampType$). here is the code: public StructType createStructType() { List<StructField> fields = new ArrayList<StructField>(); fields.add(DataType.createStructField("timestamp", DataType.TimestampType, false)); return DataType.createStructType(fields); } public void testJsonStruct(SparkContext sc) { JavaSQLContext sqlCtx = getSqlCtx(getCtx(sc)); String path = "/hdd/spark/test.json"; JavaSchemaRDD test = sqlCtx.jsonFile(path, createStructType()); sqlCtx.registerRDDAsTable(test, "test"); execSql(sqlCtx, "select * from test", 1); } Input file has a single record: {"timestamp":"2014-10-10T01:01:01"} Thanks Tridib -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-sql-timestamp-in-json-fails-tp16864.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org<mailto:user-unsubscr...@spark.apache.org> For additional commands, e-mail: user-h...@spark.apache.org<mailto:user-h...@spark.apache.org>
