Sorry for the noise; but I just remembered you're actually using MapR (and not HDFS), so maybe the "3777" trick could work...
On Thu, Jan 8, 2015 at 10:32 AM, Marcelo Vanzin <van...@cloudera.com> wrote: > Nevermind my last e-mail. HDFS complains about not understanding "3777"... > > On Thu, Jan 8, 2015 at 9:46 AM, Marcelo Vanzin <van...@cloudera.com> wrote: >> Hmm. Can you set the permissions of "/apps/spark/historyserver/logs" >> to 3777? I'm not sure HDFS respects the group id bit, but it's worth a >> try. (BTW that would only affect newly created log directories.) >> >> On Thu, Jan 8, 2015 at 1:22 AM, <michael.engl...@nomura.com> wrote: >>> Hi Vanzin, >>> >>> I am using the MapR distribution of Hadoop. The history server logs are >>> created by a job with the permissions: >>> >>> drwxrwx--- - <myusername> <mygroup> 2 2015-01-08 09:14 >>> /apps/spark/historyserver/logs/spark-1420708455212 >>> >>> However, the permissions of the higher directories are mapr:mapr and the >>> user that runs Spark in our case is a unix ID called mapr (in the mapr >>> group). Therefore, this can't read my job event logs as shown above. >>> >>> >>> Thanks, >>> Michael >>> >>> >>> -----Original Message----- >>> From: Marcelo Vanzin [mailto:van...@cloudera.com] >>> Sent: 07 January 2015 18:10 >>> To: England, Michael (IT/UK) >>> Cc: user@spark.apache.org >>> Subject: Re: Spark History Server can't read event logs >>> >>> The Spark code generates the log directory with "770" permissions. On top >>> of that you need to make sure of two things: >>> >>> - all directories up to /apps/spark/historyserver/logs/ are readable by the >>> user running the history server >>> - the user running the history server belongs to the group that owns >>> /apps/spark/historyserver/logs/ >>> >>> I think the code could be more explicitly about setting the group of the >>> generated log directories and files, but if you follow the two rules above >>> things should work. Also, I recommend setting >>> /apps/spark/historyserver/logs/ itself to "1777" so that any user can >>> generate logs, but only the owner (or a superuser) can delete them. >>> >>> >>> >>> On Wed, Jan 7, 2015 at 7:45 AM, <michael.engl...@nomura.com> wrote: >>>> Hi, >>>> >>>> >>>> >>>> When I run jobs and save the event logs, they are saved with the >>>> permissions of the unix user and group that ran the spark job. The >>>> history server is run as a service account and therefore can’t read the >>>> files: >>>> >>>> >>>> >>>> Extract from the History server logs: >>>> >>>> >>>> >>>> 2015-01-07 15:37:24,3021 ERROR Client >>>> fs/client/fileclient/cc/client.cc:1009 >>>> Thread: 1183 User does not have access to open file >>>> /apps/spark/historyserver/logs/spark-1420644521194 >>>> >>>> 15/01/07 15:37:24 ERROR ReplayListenerBus: Exception in parsing Spark >>>> event log >>>> /apps/spark/historyserver/logs/spark-1420644521194/EVENT_LOG_1 >>>> >>>> org.apache.hadoop.security.AccessControlException: Open failed for file: >>>> /apps/spark/historyserver/logs/spark-1420644521194/EVENT_LOG_1, error: >>>> Permission denied (13) >>>> >>>> >>>> >>>> Is there a setting which I can change that allows the files to be >>>> world readable or at least by the account running the history server? >>>> Currently, the job appears in the History Sever UI but only states ‘<Not >>>> Started>’. >>>> >>>> >>>> >>>> Thanks, >>>> >>>> Michael >>>> >>>> >>>> This e-mail (including any attachments) is private and confidential, >>>> may contain proprietary or privileged information and is intended for >>>> the named >>>> recipient(s) only. Unintended recipients are strictly prohibited from >>>> taking action on the basis of information in this e-mail and must >>>> contact the sender immediately, delete this e-mail (and all >>>> attachments) and destroy any hard copies. Nomura will not accept >>>> responsibility or liability for the accuracy or completeness of, or >>>> the presence of any virus or disabling code in, this e-mail. If >>>> verification is sought please request a hard copy. Any reference to >>>> the terms of executed transactions should be treated as preliminary only >>>> and subject to formal written confirmation by Nomura. >>>> Nomura reserves the right to retain, monitor and intercept e-mail >>>> communications through its networks (subject to and in accordance with >>>> applicable laws). No confidentiality or privilege is waived or lost by >>>> Nomura by any mistransmission of this e-mail. Any reference to >>>> "Nomura" is a reference to any entity in the Nomura Holdings, Inc. >>>> group. Please read our Electronic Communications Legal Notice which forms >>>> part of this e-mail: >>>> http://www.Nomura.com/email_disclaimer.htm >>> >>> >>> >>> -- >>> Marcelo >>> >>> >>> This e-mail (including any attachments) is private and confidential, may >>> contain proprietary or privileged information and is intended for the named >>> recipient(s) only. Unintended recipients are strictly prohibited from >>> taking action on the basis of information in this e-mail and must contact >>> the sender immediately, delete this e-mail (and all attachments) and >>> destroy any hard copies. Nomura will not accept responsibility or liability >>> for the accuracy or completeness of, or the presence of any virus or >>> disabling code in, this e-mail. If verification is sought please request a >>> hard copy. Any reference to the terms of executed transactions should be >>> treated as preliminary only and subject to formal written confirmation by >>> Nomura. Nomura reserves the right to retain, monitor and intercept e-mail >>> communications through its networks (subject to and in accordance with >>> applicable laws). No confidentiality or privilege is waived or lost by >>> Nomura by any mistransmission of this e-mail. Any reference to "Nomura" is >>> a reference to any entity in the Nomura Holdings, Inc. group. Please read >>> our Electronic Communications Legal Notice which forms part of this e-mail: >>> http://www.Nomura.com/email_disclaimer.htm >>> >> >> >> >> -- >> Marcelo > > > > -- > Marcelo -- Marcelo --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
