The Spark code generates the log directory with "770" permissions. On top of that you need to make sure of two things:
- all directories up to /apps/spark/historyserver/logs/ are readable by the user running the history server - the user running the history server belongs to the group that owns /apps/spark/historyserver/logs/ I think the code could be more explicitly about setting the group of the generated log directories and files, but if you follow the two rules above things should work. Also, I recommend setting /apps/spark/historyserver/logs/ itself to "1777" so that any user can generate logs, but only the owner (or a superuser) can delete them. On Wed, Jan 7, 2015 at 7:45 AM, <michael.engl...@nomura.com> wrote: > Hi, > > > > When I run jobs and save the event logs, they are saved with the permissions > of the unix user and group that ran the spark job. The history server is run > as a service account and therefore can’t read the files: > > > > Extract from the History server logs: > > > > 2015-01-07 15:37:24,3021 ERROR Client fs/client/fileclient/cc/client.cc:1009 > Thread: 1183 User does not have access to open file > /apps/spark/historyserver/logs/spark-1420644521194 > > 15/01/07 15:37:24 ERROR ReplayListenerBus: Exception in parsing Spark event > log /apps/spark/historyserver/logs/spark-1420644521194/EVENT_LOG_1 > > org.apache.hadoop.security.AccessControlException: Open failed for file: > /apps/spark/historyserver/logs/spark-1420644521194/EVENT_LOG_1, error: > Permission denied (13) > > > > Is there a setting which I can change that allows the files to be world > readable or at least by the account running the history server? Currently, > the job appears in the History Sever UI but only states ‘<Not Started>’. > > > > Thanks, > > Michael > > > This e-mail (including any attachments) is private and confidential, may > contain proprietary or privileged information and is intended for the named > recipient(s) only. Unintended recipients are strictly prohibited from taking > action on the basis of information in this e-mail and must contact the > sender immediately, delete this e-mail (and all attachments) and destroy any > hard copies. Nomura will not accept responsibility or liability for the > accuracy or completeness of, or the presence of any virus or disabling code > in, this e-mail. If verification is sought please request a hard copy. Any > reference to the terms of executed transactions should be treated as > preliminary only and subject to formal written confirmation by Nomura. > Nomura reserves the right to retain, monitor and intercept e-mail > communications through its networks (subject to and in accordance with > applicable laws). No confidentiality or privilege is waived or lost by > Nomura by any mistransmission of this e-mail. Any reference to "Nomura" is a > reference to any entity in the Nomura Holdings, Inc. group. Please read our > Electronic Communications Legal Notice which forms part of this e-mail: > http://www.Nomura.com/email_disclaimer.htm -- Marcelo --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
