Eran Medan created SPARK-5023:
---------------------------------
Summary: In Web UI job history, the total job duration is much
smaller than the sum of its stages
Key: SPARK-5023
URL: https://issues.apache.org/jira/browse/SPARK-5023
Project: Spark
Issue Type: Bug
Components: Web UI
Affects Versions: 1.2.0, 1.1.1
Environment: Amazon EC2 AMI r3.2xlarge, cluster of 20 to 50 nodes,
running the ec2 provided scripts to create.
Reporter: Eran Medan
I'm running a long process using Spark + Graph and things look good on the 4040
job status UI, but when the job is done, when going to the history then the job
total duration is much, much smaller than the total of its stages.
The way I set logs up is this:
val homeDir = sys.props("user.home")
val logsPath = new File(homeDir,"sparkEventLogs")
val conf = new SparkConf().setAppName("...")
conf.set("spark.eventLog.enabled", "true").set("spark.eventLog.dir",
logsPath.getCanonicalPath)
for example job ID X - duration 0.2 s, but when I click the job and look at its
stages, the sum of their duration is more than 15 minutes!
(before the job was over, in the 4040 job status, the job duration was correct,
it is only incorrect when its done and going to the logs)
I hope I didn't configure something because I was very surprised no one
reported it yet (I searched, but perhaps I missed it)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
