For Spark Streaming Apps, what does "Output Op Duration" in the batch details UX signify? We have been observing that - for the given batch's last output Op id - Output Op duration > Job duration by a factor. Sometimes it is huge (1 min). I have provided the screenshot below where - you can see that for Output Op Id 5, Job Duration is 59ms but output Op duration is 19s. There is no other info on where the extra 18.9 seconds were spent. On clicking the JobId - there is no breakdown on how this extra time is spent.
<http://apache-spark-user-list.1001560.n3.nabble.com/file/n28354/outputopduration.png> -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Question-about-Output-Op-Duration-in-SparkStreaming-Batch-details-UX-tp28354.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
