Hi guys,
We were able to submit our spark application to yarn's resource manager
(PivotalHD 1.0.1), and the expected result is printed in the stdout of the
container log; however, when we took a look at Tracking UI, we got the
following error 500. There is no problem for traditional mapreduce job. Is
there any suggestion to dig into this problem?
Secondly, we submitted our job through our java app, and since the
application is actually running on the remote machine, it seems that there
is no easy way to interact between our main application running locally,
and the spark application running remotely. For example, if we would like
to know the progress of our spark application, we have to write it to HDFS,
and then read it back in our main application. It's the same to retrieve
the final spark job return. Is there any way to interact through some kind
of api without going through the HDFS?
Finally, sometimes, we would like to test some code quickly through
spark-shell. And it seems that every operation is running locally when we
lunch spark-shell in yarn-client mode (I knew it's not supported to run
spark-shell in yarn-standalone mode). Is there anyway to make spark-shell
running in the distributed fashion with yarn?
Thanks.
HTTP ERROR 500
Problem accessing /proxy/application_1389853114516_1011/. Reason:
Server Error
Caused by:
java.io.IOException: java.net.URISyntaxException: Expected authority
at index 7: http://
at
org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.doGet(WebAppProxyServlet.java:318)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:652)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1320)
Sincerely,
DB Tsai
Machine Learning Engineer
Alpine Data Labs
--------------------------------------
Web: http://alpinenow.com/
