If your job involves a shuffle then the compute for the entire batch will increase with network latency. What would be interesting is to see how much time each task/job/stage takes. On Thu, Sep 22, 2016 at 5:11 PM Peter Figliozzi <pete.figlio...@gmail.com> wrote:
> It seems to me they must communicate for joins, sorts, grouping, and so > forth, where the original data partitioning needs to change. You could > repeat your experiment for different code snippets. I'll bet it depends on > what you do. > > On Thu, Sep 22, 2016 at 8:54 AM, gusiri <dreame...@gmail.com> wrote: > >> Hi, >> >> When I increase the network latency among spark nodes, >> >> I see compute time (=executor computing time in Spark Web UI) also >> increases. >> >> In the graph attached, left = latency 1ms vs right = latency 500ms. >> >> Is there any communication between worker and driver/master even 'during' >> executor computing? or any idea on this result? >> >> >> < >> http://apache-spark-user-list.1001560.n3.nabble.com/file/n27779/Screen_Shot_2016-09-21_at_5.png >> > >> >> >> >> >> >> Thank you very much in advance. >> >> //gusiri >> >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Is-executor-computing-time-affected-by-network-latency-tp27779.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >> >> >
