[
https://issues.apache.org/jira/browse/SPARK-1132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14537292#comment-14537292
]
sagar commented on SPARK-1132:
------------------------------
Hi Team,
I see the issue is resolved and Fix Version/s: is -1.0.0.
Is 1.0.0 is spark version ?
Where i can get the spark version 1.0.0.
Currently i am getting below error -
15/05/10 08:42:20 INFO deprecation: mapred.tip.id is deprecated. Instead, use
mapreduce.task.id
15/05/10 08:42:20 INFO deprecation: mapred.task.id is deprecated. Instead, use
mapreduce.task.attempt.id
15/05/10 08:42:20 INFO deprecation: mapred.task.is.map is deprecated. Instead,
use mapreduce.task.ismap
15/05/10 08:42:20 INFO deprecation: mapred.task.partition is deprecated.
Instead, use mapreduce.task.partition
15/05/10 08:42:20 INFO deprecation: mapred.job.id is deprecated. Instead, use
mapreduce.job.id
15/05/10 08:42:20 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1750
bytes result sent to driver
15/05/10 08:42:20 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0)
in 207 ms on localhost (1/1)
15/05/10 08:42:20 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have
all completed, from pool
15/05/10 08:42:20 INFO DAGScheduler: Stage 0 (count at SparkFilter.java:22)
finished in 0.225 s
15/05/10 08:42:20 INFO DAGScheduler: Job 0 finished: count at
SparkFilter.java:22, took 0.314437 s
0
15/05/10 08:42:20 ERROR LiveListenerBus: Listener EventLoggingListener threw an
exception
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:144)
at
org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:144)
at scala.Option.foreach(Option.scala:236)
at
org.apache.spark.scheduler.EventLoggingListener.logEvent(EventLoggingListener.scala:144)
at
org.apache.spark.scheduler.EventLoggingListener.onStageCompleted(EventLoggingListener.scala:165)
at
org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:32)
at
org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
at
org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
at
org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:53)
at
org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:36)
at
org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:76)
at
org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
at
org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
at
org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:60)
Caused by: java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:792)
at
org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:1998)
at
org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:1959)
at
org.apache.hadoop.fs.FSDataOutputStream.hflush(FSDataOutputStream.java:130)
... 19 more
> Persisting Web UI through refactoring the SparkListener interface
> -----------------------------------------------------------------
>
> Key: SPARK-1132
> URL: https://issues.apache.org/jira/browse/SPARK-1132
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core, Web UI
> Affects Versions: 0.9.0
> Reporter: Andrew Or
> Assignee: Andrew Or
> Priority: Blocker
> Fix For: 1.0.0
>
>
> This issue is a spin-off from another issue -
> https://spark-project.atlassian.net/browse/SPARK-969
> The main issue with the existing Spark Web UI is that its information is lost
> as soon as the application terminates. This is the direct result of the
> SparkUI being coupled with SparkContext, which is stopped when the
> application is finished.
> The attached document proposes to tackle this by logging SparkListenerEvents
> to persist information displayed on the Web UI. We take this opportunity to
> replace the existing format for storing this information, HTML, with one that
> is more flexible, JSON. This allows further post-hoc analysis of a particular
> Spark application beyond simply reviving the Web UI.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
