Could you print the time on the driver (that is, in foreachRDD but before
RDD.foreachPartition) and see if it is behaving weird?
TD
On Fri, Jun 26, 2015 at 3:57 PM, Emrehan Tüzün emrehan.tu...@gmail.com
wrote:
On Fri, Jun 26, 2015 at 12:30 PM, Sea 261810...@qq.com wrote:
Hi, all
I find
...@gmail.com; Sea261810...@qq.com;
devdev@spark.apache.org; useru...@spark.apache.org;
: Re: Time is ugly in Spark Streaming
Java's SimpleDateFormat is not thread safe. You can consider using
DateTimeFormatter if you are using Java 8 or Joda-time
On Sat, Jun 27, 2015 at 3:32 AM, Tathagata Das
Java's SimpleDateFormat is not thread safe. You can consider using
DateTimeFormatter if you are using Java 8 or Joda-time
On Sat, Jun 27, 2015 at 3:32 AM, Tathagata Das t...@databricks.com wrote:
Could you print the time on the driver (that is, in foreachRDD but
before RDD.foreachPartition)
On Fri, Jun 26, 2015 at 12:30 PM, Sea 261810...@qq.com wrote:
Hi, all
I find a problem in spark streaming, when I use the time in function
foreachRDD... I find the time is very interesting.
val messages = KafkaUtils.createDirectStream[String, String, StringDecoder,
StringDecoder](ssc,
Hi, all
I find a problem in spark streaming, when I use the time in function
foreachRDD... I find the time is very interesting.
val messages = KafkaUtils.createDirectStream[String, String, StringDecoder,
StringDecoder](ssc, kafkaParams, topicsSet)
dataStream.map(x = createGroup(x._2,
Are you sharing the SimpleDateFormat instance? This looks a lot more like
the non-thread-safe behaviour of SimpleDateFormat (that has claimed many
unsuspecting victims over the years), than any 'ugly' Spark Streaming. Try
writing the timestamps in millis to Kafka and compare.
-kr, Gerard.
On
Yes, I make it.
-- --
??: Gerard Maas;gerard.m...@gmail.com;
: 2015??6??26??(??) 5:40
??: Sea261810...@qq.com;
: useru...@spark.apache.org; devdev@spark.apache.org;
: Re: Time is ugly in Spark Streaming
Are you
