[
https://issues.apache.org/jira/browse/SPARK-5023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14512869#comment-14512869
]
Eran Medan commented on SPARK-5023:
-----------------------------------
I don't think this is a duplicate, the information shows correctly in "live"
view, the incorrect numbers are for the history / event view.
I had no lost partitions and no failures, but still - in live view, something
that took seconds or minutes shows as milliseconds or seconds in the history
view. I'll debug it and see if I can figure out the root cause.
No error messages.
> In Web UI job history, the total job duration is incorrect (much smaller than
> the sum of its stages)
> ----------------------------------------------------------------------------------------------------
>
> Key: SPARK-5023
> URL: https://issues.apache.org/jira/browse/SPARK-5023
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 1.1.1, 1.2.0
> Environment: Amazon EC2 AMI r3.2xlarge, cluster of 20 to 50 nodes,
> running the ec2 provided scripts to create.
> Reporter: Eran Medan
>
> I'm running a long process using Spark + Graph and things look good on the
> 4040 job status UI, but when the job is done, when going to the history then
> the job total duration is much, much smaller than the total of its stages.
> The way I set logs up is this:
> val homeDir = sys.props("user.home")
> val logsPath = new File(homeDir,"sparkEventLogs")
> val conf = new SparkConf().setAppName("...")
> conf.set("spark.eventLog.enabled", "true").set("spark.eventLog.dir",
> logsPath.getCanonicalPath)
> for example job ID X - duration 0.2 s, but when I click the job and look at
> its stages, the sum of their duration is more than 15 minutes!
> (before the job was over, in the 4040 job status, the job duration was
> correct, it is only incorrect when its done and going to the logs)
> I hope I didn't configure something because I was very surprised no one
> reported it yet (I searched, but perhaps I missed it)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
