Hi Andrew, sorry for late response. Thank you very much for solving my problem. There was no APPLICATION_COMPLETE file in log directory due to not calling sc.stop() at the end of program. With stopping spark context everything works correctly, so thank you again.
Best regards, Grzegorz On Fri, Sep 5, 2014 at 8:06 PM, Andrew Or <and...@databricks.com> wrote: > Hi Grzegorz, > > Can you verify that there are "APPLICATION_COMPLETE" files in the event > log directories? E.g. Does > file:/tmp/spark-events/app-name-1234567890/APPLICATION_COMPLETE exist? If > not, it could be that your application didn't call sc.stop(), so the > "ApplicationEnd" event is not actually logged. The HistoryServer looks for > this special file to identify applications to display. You could also try > manually adding the "APPLICATION_COMPLETE" file to this directory; the > HistoryServer should pick this up and display the application, though the > information displayed will be incomplete because the log did not capture > all the events (sc.stop() does a final close() on the file written). > > Andrew > > > 2014-09-05 1:50 GMT-07:00 Grzegorz Białek <grzegorz.bia...@codilime.com>: > > Hi Andrew, >> >> thank you very much for your answer. Unfortunately it still doesn't work. >> I'm using Spark 1.0.0, and I start history server running >> sbin/start-history-server.sh <dir>, although I also set >> SPARK_HISTORY_OPTS=-Dspark.history.fs.logDirectory in >> conf/spark-env.sh. I tried also other dir than /tmp/spark-events which >> have all possible permissions enabled. Also adding file: (and file://) >> didn't help - history server still shows: >> History Server >> Event Log Location: file:/tmp/spark-events/ >> No Completed Applications Found. >> >> Best regards, >> Grzegorz >> >> >> On Thu, Sep 4, 2014 at 8:20 PM, Andrew Or <and...@databricks.com> wrote: >> >>> Hi Grzegorz, >>> >>> Sorry for the late response. Unfortunately, if the Master UI doesn't >>> know about your applications (they are "completed" with respect to a >>> different Master), then it can't regenerate the UIs even if the logs exist. >>> You will have to use the history server for that. >>> >>> How did you start the history server? If you are using Spark <=1.0, you >>> can pass the directory as an argument to the sbin/start-history-server.sh >>> script. Otherwise, you may need to set the following in your >>> conf/spark-env.sh to specify the log directory: >>> >>> export >>> SPARK_HISTORY_OPTS=-Dspark.history.fs.logDirectory=/tmp/spark-events >>> >>> It could also be a permissions thing. Make sure your logs in >>> /tmp/spark-events are accessible by the JVM that runs the history server. >>> Also, there's a chance that "/tmp/spark-events" is interpreted as an HDFS >>> path depending on which Spark version you're running. To resolve any >>> ambiguity, you may set the log path to "file:/tmp/spark-events" instead. >>> But first verify whether they actually exist. >>> >>> Let me know if you get it working, >>> -Andrew >>> >>> >>> >>> 2014-08-19 8:23 GMT-07:00 Grzegorz Białek <grzegorz.bia...@codilime.com> >>> : >>> >>> Hi, >>>> Is there any way view history of applications statistics in master ui >>>> after restarting master server? I have all logs ing /tmp/spark-events/ but >>>> when I start history server in this directory it says "No Completed >>>> Applications Found". Maybe I could copy this logs to dir used by master >>>> server but I couldn't find any. Or maybe I'm doing something wrong >>>> launching history server. >>>> Do you have any idea how to solve it? >>>> >>>> Thanks, >>>> Grzegorz >>>> >>>> >>>> On Thu, Aug 14, 2014 at 10:53 AM, Grzegorz Białek < >>>> grzegorz.bia...@codilime.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> Thank you both for your answers. Browsing using Master UI works fine. >>>>> Unfortunately History Server shows "No Completed Applications Found" even >>>>> if logs exists under given directory, but using Master UI is enough for >>>>> me. >>>>> >>>>> Best regards, >>>>> Grzegorz >>>>> >>>>> >>>>> >>>>> On Wed, Aug 13, 2014 at 8:09 PM, Andrew Or <and...@databricks.com> >>>>> wrote: >>>>> >>>>>> The Spark UI isn't available through the same address; otherwise new >>>>>> applications won't be able to bind to it. Once the old application >>>>>> finishes, the standalone Master renders the after-the-fact application UI >>>>>> and exposes it under a different URL. To see this, go to the Master UI >>>>>> (<master-url>:8080) and click on your application in the "Completed >>>>>> Applications" table. >>>>>> >>>>>> >>>>>> 2014-08-13 10:56 GMT-07:00 Matei Zaharia <matei.zaha...@gmail.com>: >>>>>> >>>>>> Take a look at http://spark.apache.org/docs/latest/monitoring.html >>>>>>> -- you need to launch a history server to serve the logs. >>>>>>> >>>>>>> Matei >>>>>>> >>>>>>> On August 13, 2014 at 2:03:08 AM, grzegorz-bialek ( >>>>>>> grzegorz.bia...@codilime.com) wrote: >>>>>>> >>>>>>> Hi, >>>>>>> I wanted to access Spark web UI after application stops. I set >>>>>>> spark.eventLog.enabled to true and logs are availaible >>>>>>> in JSON format in /tmp/spark-event but web UI isn't available under >>>>>>> address >>>>>>> http://<driver-node>:4040 >>>>>>> I'm running Spark in standalone mode. >>>>>>> >>>>>>> What should I do to access web UI after application ends? >>>>>>> >>>>>>> Thanks, >>>>>>> Grzegorz >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> View this message in context: >>>>>>> http://apache-spark-user-list.1001560.n3.nabble.com/Viewing-web-UI-after-fact-tp12023.html >>>>>>> Sent from the Apache Spark User List mailing list archive at >>>>>>> Nabble.com. >>>>>>> >>>>>>> --------------------------------------------------------------------- >>>>>>> >>>>>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>>>>>> For additional commands, e-mail: user-h...@spark.apache.org >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
