I tried everything including sudo, but it still did not work using the local directory. However, I finally got it working by getting the history server to log into hdfs. I first created a directory in hdfs like the following: ./ephemeral-hdfs/bin/hadoop fs -mkdir /spark_logs Then I started the started the history server like the following.
./start-history-server.sh hdfs:///spark_logs --port 18080 In order to see the history server UI I needed to open up inbound traffic for the port 18080 in AWS. As follows custom TCP port 18080 from anywhere Hope this will help others. Zhen -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/problem-starting-the-history-server-on-EC2-tp7361p7436.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
