Why Executor Deserialize Time takes more than 300ms?

In our experimental cluster (1 driver, 5 workers), we tried the simplest example:   sc.parallelize(Range(0, 100), 2).count  In the event log, we found the executor takes too much time on deserialization, about 300 ~ 500ms, and the execution time is only 1ms.  Our servers are with 2.3G Hz CPU

Re: Why Executor Deserialize Time takes more than 300ms?

Hi Xuelin, this type of question is probably better asked on the spark-user mailing list, u...@spark.apache.org http://apache-spark-user-list.1001560.n3.nabble.com Do you mean the very first set of tasks take 300 - 500 ms to deserialize? That is most likely because of the time taken to ship the

Re: Why Executor Deserialize Time takes more than 300ms?

Thanks Imran, The problems is, *every time* I run the same task, the deserialization time is around 300~500ms. I don't know if this is a normal case. -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Why-Executor-Deserialize-Time-takes-more

Why Executor Deserialize Time takes more than 300ms?

.1001551.n3.nabble.com/Why-Executor-Deserialize-Time-takes-more-than-300ms-tp9476.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org